Session Timeout Occurs When Running in Docker on Linux with High Device Count

[yonasofer]

I am using Renci.SshNet in an application that monitors networked devices. Each device executes multiple commands at different intervals (e.g., every 10 seconds, 15 seconds, and 2 minutes). Commands run concurrently across different devices, but a maximum of 5 commands run in parallel per device.

The issue arises when monitoring a higher number of devices (approximately 30 or more). In this scenario, I start receiving widespread session timeout errors across all devices, even though there are no network issues. These timeouts occur at random intervals (every few minutes). However, when monitoring a lower number of devices (around 14), everything functions without issues.

Notably, this issue only occurs when running the application as a Docker container on a Linux environment. When running the same application as a standalone .exe on Windows, the problem does not occur.

Environment Details:
• Library Version: Renci.SshNet 2024.2.0 (master)
• Also tested on: develop branch version adapted for .NET 7.0 (issue persists)
• Platform (Working Case): Windows (running as an .exe)
• Platform (Failing Case): Linux (running inside a Docker container)
• Docker Base Image: [Specify the base image if relevant, e.g., mcr.microsoft.com/dotnet/runtime:7.0]
• Number of Devices: ~30+ (issue occurs) / ~14 (no issue)
• Parallel Execution: Up to 5 commands per device
• Error Message: Session timeout

Code Snippet:

This is the code used to establish an SSH connection and execute command:

try
{
    using var client = new SshClient(Host, CycladeAdminInfo.User, CycladeAdminInfo.Password);
    client.ConnectionInfo.Timeout = TimeSpan.FromMilliseconds(30000);
    client.Connect();
    
    using var cmd = client.CreateCommand(command);
    cmd.CommandTimeout = TimeSpan.FromMilliseconds(30000);
    output = cmd.Execute();
    
    client.Disconnect();
    result = true;
}
catch (Exception ex)
{
    Console.WriteLine($"{DateTime.Now:HH:mm:ss:fff} {Host} {command} {ex.Message} {ex.StackTrace}");
}

Stack Trace:

Connection failed to establish within 30000 milliseconds.
at Renci.SshNet.Abstractions.SocketAbstraction.ConnectCore (Socket socket, EndPoint remoteEndpoint, TimeSpan connectTimeout, Boolean ‹
at Renci.SshNet.Abstractions.SocketAbstraction.Connect (Socket socket, EndPoint remoteEndpoint, TimeSpan connectTimeout)
at Renci.SshNet.Connection.ConnectorBase.SocketConnect (EndPoint endPoint, TimeSpan timeout)
at Renci.SshNet.Connection.DirectConnector.Connect (IConnectionInfo connectionInfo)
at Renci.SshNet.Session.Connect ()
at Renci.SshNet.BaseClient.CreateAndConnectSession ()
at Renci.SshNet.BaseClient.Connect ()
at Communication.SshOutletCommunicator.SendStringNew (String command, Device device, Strings output)

Steps to Reproduce:
1. Deploy the application in a Docker container on a Linux environment.
2. Connect to 30+ devices and execute multiple SSH commands in parallel.
3. Observe that after a few minutes, session timeouts occur across all devices.
4. Run the same application on Windows as an .exe and confirm that the issue does not occur.

Expected Behavior:

The SSH sessions should remain stable and function correctly regardless of whether the application is running on Windows or Linux in Docker.

Actual Behavior:

When running in Docker on Linux, widespread session timeouts occur at random intervals when monitoring a large number of devices.

Additional Notes:
• There are no observed network issues.
• The issue is consistent across multiple test environments.
• It may be related to how Renci.SshNet handles sessions in a Linux/Docker context.

Any guidance or suggestions for troubleshooting would be greatly appreciated. Thank you!``

[Rob-Hague]

The error is occurring during the initial socket connect. I imagine you might see the same problem just with a socket and no library code e.g. new Socket(SocketType.Stream, ProtocolType.Tcp).Connect(Host, 22); ?

I think for troubleshooting your best bet will be to look at packet captures