Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable multisharding for solid queue #505

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

alxckn
Copy link

@alxckn alxckn commented Jan 30, 2025

Fixes #353

This is a proposal to add multisharding as a supported configuration option for Solid Queue. It makes it possible to easily migrate incrementally an application from one to multiple shards as changing the connects_to configuration will not change any of the default behavior of Solid Queue.

This introduces 2 ways to determine which shard to enqueue a job to:

  • specify a configured queue_adapter on a job, so that a given job class would be assigned to a given shard (ensuring full compatibility with some other Solid Queue features such as concurrency control)
  • specify a global shard_selection_lambda to have complete control over shard selection logic when enqueueing jobs

By configuring the active_shard dynamically (for instance using an environment variable), it is possible to start workers, dispatchers and schedulers that will consume each shard.

@alxckn alxckn marked this pull request as ready for review January 30, 2025 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

DB Shards support
1 participant