Don't signal shutdown Cancellation Token when starting Drain Mode and include end to end tests

[nzthiago]

Customers don't expect a shut down Cancellation Token (related: #2782) when the host enters drain mode.

To ensure currently executing .NET in process functions are allowed to complete successfully we should not signal a host shutdown cancellation token when entering drain mode.

Also as part of the Drain Mode End to End Tests we should have one or more end to end tests that has:

• A long running function that doesn't have a cancellation token in its declaration and a long running function that does have a cancellation token in its declaration
• one or multiple function invocations are started for both functions (with and without cancellation token), then enables Drain Mode, then ensures those functions are allowed to complete successfully

Tasks

• Create issue in the Kafka WebJobs extension drain mode/cancellation investigation & changes
  • Do not signal cancellation when in drain mode azure-functions-kafka-extension#530
• Create issue in the SignalR WebJobs extension drain mode/cancellation investigation & changes
  • [SignalR] Do not signal cancellation when in drain mode azure-sdk-for-net#47420
• Create issue in the RabbitMQ WebJobs extension drain mode/cancellation investigation & changes
  • Do not signal cancellation when in drain mode azure-functions-rabbitmq-extension#241

Status (12/06/2024)

• The following extensions have fixed this issue: Cosmos, Storage, ServiceBus, EventHub, Timer
• EventGrid and SendGrid were investigated and do NOT require any changes/fixes to be made.
• GH issues were raised for these extensions to address the cancellation/drain mode issue: RabbitMQ, SignalR, and Kafka.

[liliankasem]

Potentially related to #2776

[es-alt]

@soninaren , what nuget version is this in?

[soninaren]

@soninaren , what nuget version is this in?

Should be version 3.0.33. I don't think it has been released yet.

[es-alt]

@soninaren , this affects us a lot in production with HttpTriggered functions. Is there at least an approximate estimate of when 3.0.33 can be released?

[soninaren]

@LeCamarade are you using the webjobs sdk or a function app. I think the function release is already in progress.

[es-alt]

@soninaren , we use a function app. If release is in progress, are we talking, days, weeks or months?

[soninaren]

@soninaren , we use a function app. If release is in progress, are we talking, days, weeks or months?

Based on the current release progress it looks like it should be complete in a couple of weeks.

[mathewc]

Reactivating this because we're seeing inconsistent behavior across various extensions when drain is initiated. For example, the latest versions of both the Azure Queues and ServiceBus extensions trigger the cancellation token when drain is started.

[drdamour]

i've been using azure functions for going on 4 years and drain mode is still a mystery to me. I had incorrectly assumed that it WOULD send cancellation tokens and got really confused when it did for some extensions and not for others. Seems like some docs around drain mode are missing in the az function space and it should be encouraged for extensions to document their expected drain mode behaviour

the recent "fix" in event hub extension pointed out IDrainModeManager which i had never come across in the Az functions docs.

limited results from google on this, some appear to be completely wrong like https://learn.microsoft.com/en-us/answers/questions/770912/why-do-i-see-drainmode-mode-enabled-in-the-logs-an?source=docs

At this time, it sends a request to the Admin API to enter drain mode. Then all currently executing Functions on that instance receive a cancellation token which the customer can handle.

emphasis added.

and https://datanrg.blogspot.com/2021/06/azure-function-drain-mode.html

When the host is put in drain mode, It should:
Stop listening for new incoming requests,
Cancellation token is passed as a parameter to the function invocation,
Finally, a scale-in operation is performed.

[liliankasem]

@drdamour IDrainModeManager is for internal use and is not intended for customers to use in their applications so you will not hear about this in the docs anywhere.

The intended behaviour for drain mode is:

1. Stop listening for new incoming requests
2. Allow existing function invocations to finish executing without interruption
3. Scale-in operation is performed when all invocations are completed

I think there was some miscommunication on the intended behaviour which is why some extensions send a cancellation token and some don't, this is something that is actively being fix across the board.

[drdamour]

@liliankasem um.... then why is it being advertised for use in recent release notes?

https://github.com/Azure/azure-sdk-for-net/blob/Microsoft.Azure.WebJobs.Extensions.EventHubs_5.5.0/sdk/eventhub/Microsoft.Azure.WebJobs.Extensions.EventHubs/CHANGELOG.md

https://github.com/Azure/azure-sdk-for-net/blob/Microsoft.Azure.WebJobs.Extensions.ServiceBus_5.12.0/sdk/servicebus/Microsoft.Azure.WebJobs.Extensions.ServiceBus/CHANGELOG.md

[liliankasem]

🤷‍♀️ news to me

[drdamour]

well...if it works as advertised it certainly fulfills a need i as an app developer have, though more in the durable functions space of knowing when my instance has entered drain mode.

[nzthiago]

This is affecting customers on Flex Consumption as it is more aggressive to scale in, blocking some from going to production.

As a repro:

I have a very simple function with a delay of 20 seconds, this is the only function in the app:

It has a 20 second delay in it. It seems like drain mode is triggering the cancellation token (see in the logs that the token returns true), and this causes the message to appear in the queue again after a few minutes and this repeats again:

Requesting prioritizing fixing this across all triggers.