Enable multisharding for solid queue #505
Open
+389
−6
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #353
This is a proposal to add multisharding as a supported configuration option for Solid Queue. It makes it possible to easily migrate incrementally an application from one to multiple shards as changing the
connects_to
configuration will not change any of the default behavior of Solid Queue.This introduces 2 ways to determine which shard to enqueue a job to:
queue_adapter
on a job, so that a given job class would be assigned to a given shard (ensuring full compatibility with some other Solid Queue features such as concurrency control)shard_selection_lambda
to have complete control over shard selection logic when enqueueing jobsBy configuring the
active_shard
dynamically (for instance using an environment variable), it is possible to start workers, dispatchers and schedulers that will consume each shard.