Client to Silo communication not working, hosted on App Services Azure

[rogeralvarez-tapkey]

Hello everyone,

I am creating an Orleans-based application using a community repository as a reference.
https://github.com/bradygaster/OrleansOnAzureAppService
That repository uses .NET 6.0 and Orleans 3.5

I have a similar setup but using Orleans 7.0 and .NET 7.0 instead.
This is an image of the topology of the solution:

Most communication is already working, but the communication from the Client to Silo Server is not working.
The most important finding I have until now is that Using Orleans v3.5 ipAdress created in the Grain Membership Directory is using the vnet address but in version 7.0 it is using a public IP:

v3.5

v7.0

The error obtained is the following one:

Orleans.Runtime.Messaging.NetworkingTrace[0]
      Connection attempt to endpoint S172.17.0.42:20045:0 failed
      System.OperationCanceledException: The operation was canceled.
         at System.Threading.CancellationToken.ThrowOperationCanceledException()
         at System.Threading.CancellationToken.ThrowIfCancellationRequested()
         at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/Shared/SocketConnectionFactory.cs:line 53
         at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/ConnectionFactory.cs:line 61
         at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 193
Unhandled exception. Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to any of the 1 available gateways.
 ---> Orleans.Runtime.Messaging.ConnectionFailedException: Connection attempt to endpoint S172.17.0.42:20045:0 timed out after 00:00:05

I have been trying to use AdvertisedIPAddress but it does not work either.

This repository has the minimum setup to reproduce the issue using Orleans 7.1.1
https://github.com/rogeralvarez-tapkey/OrleansOnAzureAppService

I there any advice or suggestion you can help me with?

[bradygaster]

Adding @btardif and @IEvangelist here, as there was a recent change in App Service on Windows that impacted our guidance on deploying Orleans Apps to App Service Windows. @btardif - if this change has been documented, can we get a link to the App Service post summarizing the Orleans-impacting networking change made recently? Also @IEvangelist - we should make a big "purple box" or something in the doc to call out guidance for folks who try this out on App Service Windows.

The TL/DR here is that, post-this-change, App Services on the VNet must also be in separate App Service Plans, if running on App Service Windows. From your Bicep diagram above, it appears as though your App Services are running in the same App Service Plan, so, that means they'd be unable to communicate on those private ports until they're on separate App Service Plans. You could also change to App Service Linux, and this problem would be resolved. The issue is only resident on App Service for Windows, and at this time I don't have an update from the service team as to if there will be a change back to the old way of hosting Orleans apps prior to this networking change.

[rogeralvarez-tapkey]

Thank you so much for your quick reply @bradygaster
I have tried the potential workaround using Linux App Service instead of Windows and I am getting a similar error:

 Orleans.Runtime.Messaging.NetworkingTrace[0]
2023-04-04T17:11:18.919689145Z       Connection attempt to endpoint S169.254.130.2:20045:0 failed
2023-04-04T17:11:18.919696345Z       System.OperationCanceledException: The operation was canceled.
2023-04-04T17:11:18.920881752Z          at System.Threading.CancellationToken.ThrowOperationCanceledException()
2023-04-04T17:11:18.920919552Z          at System.Threading.CancellationToken.ThrowIfCancellationRequested()
2023-04-04T17:11:18.920925752Z          at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/Shared/SocketConnectionFactory.cs:line 53
2023-04-04T17:11:18.920930352Z          at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/ConnectionFactory.cs:line 61
2023-04-04T17:11:18.920934652Z          at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 193
2023-04-04T17:11:19.229680912Z Unhandled exception. Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to any of the 1 available gateways.
2023-04-04T17:11:19.229720012Z  ---> Orleans.Runtime.Messaging.ConnectionFailedException: Connection attempt to endpoint S169.254.130.2:20045:0 timed out after 00:00:05

Ip created into the Membership table is still not within the vnet subnet

I am attaching the reproduced repo using Linux instead of Windows
https://github.com/rogeralvarez-tapkey/OrleansOnAzureAppService/tree/feature/using-linux-app-service

Do you think this issue is going to be present using Container Apps?

[IEvangelist]

Adding @btardif and @IEvangelist here, as there was a recent change in App Service on Windows that impacted our guidance on deploying Orleans Apps to App Service Windows. @btardif - if this change has been documented, can we get a link to the App Service post summarizing the Orleans-impacting networking change made recently? Also @IEvangelist - we should make a big "purple box" or something in the doc to call out guidance for folks who try this out on App Service Windows.

The TL/DR here is that, post-this-change, App Services on the VNet must also be in separate App Service Plans, if running on App Service Windows. From your Bicep diagram above, it appears as though your App Services are running in the same App Service Plan, so, that means they'd be unable to communicate on those private ports until they're on separate App Service Plans. You could also change to App Service Linux, and this problem would be resolved. The issue is only resident on App Service for Windows, and at this time I don't have an update from the service team as to if there will be a change back to the old way of hosting Orleans apps prior to this networking change.

Hi @bradygaster,

Please see the following PRs:

PR	Description
dotnet/docs#34875	.NET Docs content update, specifically calling attention to the Azure App Service changes on Windows. Also, did an edit pass, updated ms.date and recaptured Bicep images from VS code visualizer.
Azure-Samples/Orleans-Cluster-on-Azure-App-Service#5	Sample app updates, added staging step in deploy workflow, added staging slots to Bicep deployment infrastructure, ensured GitHub Action versions are updated, upgrade to .NET 7.0, and major version API updates where applicable.

Please let me know if there's anything else to update, happy to do it.

[rogeralvarez-tapkey]

That example is running correctly, however in terms of cluster configuration in the scenario of Shopping Cars the topology is following the co-hosted client architecture, is there any example using a different app service between Server and Client @IEvangelist ?

[btardif]

@seligj95 can you help here?

[bradygaster]

Do you think this issue is going to be present using Container Apps?

I do not think this would be an issue on ACA.

[rogeralvarez-tapkey]

Hello @bradygaster I have been trying to use ACA, but faced multiple limitations on Azure,

The First limitation I have found is that the Membership table is being fulfilled using localhost IP, so from Client to Silo networking could be a challenge:

The second issue I have found is that ACA can only expose 1 port at a time, this is something that has been reported in the past but does not seem like it is going to be resolved soon:
microsoft/azure-container-apps#201

I can configure the Silo to expose 30000 port but health could be a challenge to configure there.

The repositories I am using to replicate issues are:
https://github.com/Azure-Samples/Orleans-Cluster-on-Azure-Container-Apps
Using Orleans 3.6.1

https://github.com/rogeralvarez-tapkey/OrleansOnAzureAppService/tree/feature/using-container-app-service
Using Orleans 7.1.1

Issues:

Silo is not able to communicate with Dashboard and the client is not able to connect

[seligj95]

Following up with our networking team on the Windows App Service update ETA for this. Migrating to Linux App Service would be an immediate fix with minimal cost changes if that is feasible option.

[rogeralvarez-tapkey]

I still getting similar issues when migrated to Linux App Services @seligj95 #8365 (reply in thread)

On this repository you can quickly reproduce the issue https://github.com/rogeralvarez-tapkey/OrleansOnAzureAppService/tree/feature/using-linux-app-service

[rogeralvarez-tapkey]

As a quick summary until now, using Orleans 7.0.0+ is not working when hosting clients and silos in different apps wither by using Windows App Services as well as Linux App Services, using Container App Services seems to have additional limitations and challenges due to the limit of having only one exposed port per container app.

Repositories to test with are:
Windows App Service: https://github.com/rogeralvarez-tapkey/OrleansOnAzureAppService
Linux App Service: https://github.com/rogeralvarez-tapkey/OrleansOnAzureAppService/tree/feature/using-linux-app-service
ACA:
https://github.com/Azure-Samples/Orleans-Cluster-on-Azure-Container-Apps
https://github.com/rogeralvarez-tapkey/OrleansOnAzureAppService/tree/feature/using-container-app-service