Slow client calls but fast grain execution

[NicolaAtorino]

Hello, we're using an Orleans Cluster for one of our systems, and we noticed that when the application is under load several of our calls from outside the silo see a considerable increase in latency. However, and that's the part that we're not getting, the metrics show that the actual execution of the grain method is not getting worse and it still in the single-digit milliseconds.

For some reason, there is something between the call and the execution that slows down. I was trying to look for grain activation latency metrics (in case the slowness would be imputable to the actual activation of the grain, before the code execution) but could not find anything specific in the orleans counters that would show me this information. Moreover, we are sending requests mainly to the same pool of grains, so i imagine that any possible activation latency would be an issue only the first time.

The grain is marked as re-entrant, so this should not be related to queueing of single-threaded requests.
The problem is reproducible even with a single silo in the cluster, so it should not be related to internal networking. However, strangely we noticed that increasing the number of silo makes the matter worse, without changing the rate of requests coming coming from the client application.
The CPU spike is expected due to some intensive operation. The machine, regardless of the CPU spike, still has plenty of CPU available. The thread count as well does not change during the CPU spike and remains low.

Usually these call take < 60 ms, but during a cpu spike we can see the latency going > 300ms .

I was thinking that this could be an issue related to the 'routing' of requests from the silo to the grain, but any help in how to continue the investigation would be appreciated.

[HermesNew]

May be the same issue as #8722