AutorecoveringConnection Memory Leak

[NathanielAB]

Describe the bug

Hi we have noticed a large amount of TimerQueues when using RabbitMq latest version 7.1.1, leading to an increase in memory usage

Reproduction steps

Start multiple consumers in parallel threads and by time the memory usage increases

Expected behavior

No memory leaks when connecting to multiple consumers

Additional context

No response

[lukebakken]

@NathanielAB - it would be greatly appreciated if you took the time to do the following:

• Provide code to reproduce the issue. Yes, you described how ("Start multiple consumers in parallel threads and by time the memory usage increases"), but to ensure I reproduce the issue the same way, I would appreciate code I can clone, compile and run. A single sentence can be easily mis-interpreted.
• Describe the tool you're using to track allocations, and how to use it. I can't tell from your screenshot.
• If you can, provide a PR to address the issue.

[lukebakken]

@NathanielAB I can't make progress without more information.

Related issues / comments:

• Intermittent deadlock when closing a channel using CloseAsync in 7.x #1751
• Bug in IntervalList causes leak in locks. #1784
• Intermittent deadlock when closing a channel using CloseAsync in 7.x #1751 (comment)

So, it seems that #1784 didn't really fix the issue around timer leaks.

Pinging people who worked on this issue before to see if they can contribute:

@ZajacPiotr98
@DenisMayorko

[NathanielAB]

@NathanielAB - it would be greatly appreciated if you took the time to do the following:

• Provide code to reproduce the issue. Yes, you described how ("Start multiple consumers in parallel threads and by time the memory usage increases"), but to ensure I reproduce the issue the same way, I would appreciate code I can clone, compile and run. A single sentence can be easily mis-interpreted.
• Describe the tool you're using to track allocations, and how to use it. I can't tell from your screenshot.
• If you can, provide a PR to address the issue.

Hi sorry for the late reply, I will try and produce a small POC console application.

The tool I used was visual studio I got a memory dump (.gcdump) and analyzed it there. It shows the root (multiple references nested within each other of TimerQueue) and the referenced types, the types generating the TimerQueues, mostly RabbitMq. It seems to be happening when there is a recovery of a connection, because the specific method causing these instances is RecoverConnectionAsync.

[lukebakken]

Thank you @NathanielAB. It would also be great to know if you're using .NET core or a netstandard2.0-TFM environment.

[NathanielAB]

Thank you @NathanielAB. It would also be great to know if you're using .NET core or a netstandard2.0-TFM environment.

.NET Core

[lukebakken]

@NathanielAB if you aren't able to provide code to reproduce the issue, could you log all first-chance-exceptions when this problem presents itself?

I think what may be happening is that an exception is being thrown here:

https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/main/projects/RabbitMQ.Client/Impl/AutorecoveringConnection.cs#L297-L298

...and this prevents this cancellation token source from being disposed:

https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/main/projects/RabbitMQ.Client/Impl/AutorecoveringConnection.cs#L303

I'm making some code changes and can have a 7.1.3 alpha release available soon, as well.

[lukebakken]

@NathanielAB - you can get 7.1.3-alpha.0 from here:

https://www.myget.org/feed/rabbitmq-dotnet-client/package/nuget/RabbitMQ.Client

It includes code from PR #1814.

[lukebakken]

@NathanielAB any updates from you? I'd like to wrap this issue up this week, thanks.

[NathanielAB]

Hi @lukebakken we have ran the alpha package and seems to have helped with the leak, thank you!

[lukebakken]

we have ran the alpha package and seems to have helped with the leak,

OK, by "helped" you mean it's fixed, correct?

Could you please log first-chance exceptions? I would like to know which exception is causing the existing code to skip .Dispose() here:

https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/main/projects/RabbitMQ.Client/Impl/AutorecoveringConnection.cs#L297-L298

[NathanielAB]

we have ran the alpha package and seems to have helped with the leak,

OK, by "helped" you mean it's fixed, correct?

Could you please log first-chance exceptions? I would like to know which exception is causing the existing code to skip .Dispose() here:

https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/main/projects/RabbitMQ.Client/Impl/AutorecoveringConnection.cs#L297-L298

Yes seems like it has been fixed. Will try to investigate regarding first chance exceptions

[NathanielAB]

Im only seeing the following, nothing in the Dispose method

FirstChanceException event raised in Project: System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'RabbitMQ.Client.Framing.AutorecoveringConnection'.
   at RabbitMQ.Client.Framing.AutorecoveringConnection.<ThrowIfDisposed>g__ThrowDisposed|76_0()

and

FirstChanceException event raised in Project: System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at System.Threading.CancellationToken.ThrowIfCancellationRequested()
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at RabbitMQ.Client.Impl.SocketFactory.ConnectUsingAddressFamilyAsync(IPEndPoint endpoint, Func`2 socketFactory, AddressFamily family, TimeSpan connectionTimeout, CancellationToken cancellationToken)
   at RabbitMQ.Client.Impl.SocketFactory.OpenAsync(AmqpTcpEndpoint amqpTcpEndpoint, Func`2 socketFactory, TimeSpan connectionTimeout, CancellationToken cancellationToken)
   at RabbitMQ.Client.Impl.SocketFrameHandler.CreateAsync(AmqpTcpEndpoint amqpTcpEndpoint, Func`2 socketFactory, TimeSpan connectionTimeout, CancellationToken cancellationToken)
   at RabbitMQ.Client.ConnectionFactory.CreateFrameHandlerAsync(AmqpTcpEndpoint endpoint, CancellationToken cancellationToken)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

[lukebakken]

Thank you! I will release version 7.1.3 today.