feat(taskworker): Remove task input queue #86151
Draft
+221
−137
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It was found that tasks were dying in the queue while waiting to be executed. One workaround for this is to have the input queue be only of size 1, but that means there is still one task that might die in the queue waiting to be executed. Instead, have the children signal to the parent that they are ready for more tasks. This way the parent doesn't make any gRPC calls unless the children are idle. Change the children to also pass one end of a pipe to the parent in the results queue. The parent is constantly draining the queue, and for every result in the queue, is fetching a new task and putting it on one of the channels. This is also a first step towards making batched gRPC calls, since now the parent is aware of how many tasks it needs to update/fetch at a time. The size of the queue can be set to the number of children, and when a child starts up it will put a signal into the queue (WaitResult) that is just a marker that the child is ready for a task. In this scenario, in order to avoid constant polling of the broker, if the queue is full of idle workers, and no new tasks are fetched, then backoff.
github-actions
bot
added
the
Scope: Backend
❌ 8 Tests Failed:
View the top 3 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It was found that tasks were dying in the queue while waiting to be executed. One workaround for this is to have the input queue be only of size 1, but that means there is still one task that might die in the queue waiting to be executed.
Instead, have the children signal to the parent that they are ready for more tasks. This way the parent doesn't make any gRPC calls unless the children are idle.
Change the children to also pass one end of a pipe to the parent in the results queue. The parent is constantly draining the queue, and for every result in the queue, is fetching a new task and putting it on one of the channels.
This is also a first step towards making batched gRPC calls, since now the parent is aware of how many tasks it needs to update/fetch at a time.
The size of the queue can be set to the number of children, and when a child starts up it will put a signal into the queue (WaitResult) that is just a marker that the child is ready for a task.
In this scenario, in order to avoid constant polling of the broker, if the queue is full of idle workers, and no new tasks are fetched, then backoff.