Dedicated queue writers #5029
Open
Dedicated queue writers #5029
+523
−885
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
roborivers
suggested changes
Feb 22, 2025
There was a problem hiding this comment.
Coding style check: Error. ⚠.
Smoke testing: Success ✓.
Cbuild submission: Success ✓.
Regression testing: 3/598 tests failed ⚠.
The first 10 failing tests are:
osql_cleanup
reco-ddlk-sql
incremental_backup_load
Increasing the number of writers on the master does not always improve database performance. More often than not it increases contention and hurts performance. But queues may be an exception: only one client can consume from a queue at a time, that it could make sense for queue consumers to run in parellel as much as possible. On a tunable, this patch dispatches queue consumes to dedicated writer threads. The handle_buf code is also refactored to use thdpool. Signed-off-by: Rivers Zhang <[email protected]>
riverszhang89
force-pushed
the
handle_buf_qpool
branch
from
5a8c763 to
1af0483
Compare
roborivers
suggested changes
Feb 24, 2025
There was a problem hiding this comment.
Coding style check: Error. ⚠.
Smoke testing: Success ✓.
Cbuild submission: Error ⚠.
Regression testing: 3/598 tests failed ⚠.
The first 10 failing tests are:
triggersc_latency
reco-ddlk-sql
incremental_backup_load
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Increasing the number of writers on the master does not always improve database performance. More often than not it increases contention and hurts performance. But queues may be an exception: only one client can consume from a queue at a time, that it could make sense for queue consumers to run in parellel as much as possible.
On a tunable, this patch dispatches queue consumes to dedicated writer threads. The handle_buf code is also refactored to use thdpool.