rpc timeout when load increases on kubernetes cluster

[kksuen]

I am doing load testing on kubernetes cluster and found that when it reaches certain load, all request after that will fail with Timeout waiting for RPC response. This happens after 2 mins of waiting.
After some search found out that there's a setting in ocr_rpc_client.go.
RPC_RESPONSE_TIMEOUT = time.Second * 120

Can this be changed in without re-compiling the docker image or is there a problem with the setup of kubernetes or is there a problem' with the underlying codes?

If i understand correctly all the worker threads/pods wiff process all the queue request without and returns the result to the httpd and in turn returns the response to the user. I think 2 mins is a bit short.