Skip to content

Commit be7b80e

Browse files
committed
Add documentation
1 parent 10248b8 commit be7b80e

File tree

1 file changed

+67
-0
lines changed

1 file changed

+67
-0
lines changed

README.md

+67
Original file line numberDiff line numberDiff line change
@@ -609,6 +609,73 @@ my_periodic_resque_job:
609609

610610
and the job will be enqueued via `perform_later` so it'll run in Resque. However, in this case we won't track any `solid_queue_recurring_execution` record for it and there won't be any guarantees that the job is enqueued only once each time.
611611

612+
## Multisharding
613+
614+
If your application reaches a point where the pressure on the database used for jobs is such that you need to spread the load over multiple databases, then this section is for you.
615+
616+
You can extend the Solid Queue database configuration to use different shards:
617+
618+
```ruby
619+
config.solid_queue.connects_to = {
620+
shards: {
621+
queue_shard_one: { writing: :queue_shard_one },
622+
queue_shard_two: { writing: :queue_shard_two }
623+
}
624+
}
625+
```
626+
627+
Queue database shards will need to have been defined in `config/database.yml` as shown in the installation section. Both shards need to share the same schema, and down the line share the same migration configuration:
628+
629+
```yaml
630+
production:
631+
primary:
632+
<<: *default
633+
database: storage/production.sqlite3
634+
queue_shard_one:
635+
<<: *default
636+
database: storage/production_queue_shard_one.sqlite3
637+
migrations_paths: db/queue_migrate
638+
queue_shard_two:
639+
<<: *default
640+
database: storage/production_queue_shard_two.sqlite3
641+
migrations_paths: db/queue_migrate
642+
```
643+
644+
Simply converting a simpler database configuration such as `config.solid_queue.connects_to = { database: { writing: :queue } }` to `config.solid_queue.connects_to = { shards: { queue: { writing: :queue } } }` will not have any effects on the behavior of Solid Queue.
645+
646+
### Configuration
647+
648+
In `config/environments/production.rb` or for the environment you want to enable Solid Queue in, you can define the following options:
649+
650+
```ruby
651+
config.solid_queue.primary_shard = :queue_shard_one # optional
652+
config.solid_queue.active_shard = ENV["SOLID_QUEUE_ACTIVE_SHARD"]&.to_sym
653+
config.solid_queue.shard_selection_lambda = ->(active_job:, active_jobs:) { nil }
654+
```
655+
656+
- `config.solid_queue.primary_shard` is the shard that will be used to enqueue or schedule jobs without any specific adapter configuration. It defaults to the first shard found in `config.solid_queue.connects_to` (ActiveRecord default)
657+
- `config.solid_queue.active_shard` is the shard that will be used by workers, dispatchers and schedulers to manage and process jobs. It defaults to the `primary_shard`.
658+
With a basic Solid Queue configuration and the option described above you can start a worker and dispatcher working on the `queue_shard_two` shard by running `SOLID_QUEUE_ACTIVE_SHARD=queue_shard_two bin/jobs start`
659+
- `config.solid_queue.shard_selection_lambda` helps you define a custom strategy to determine in which shard a job should be enqueued. It accepts keyword parameters `active_job` when a single job is enqueued or scheduled and `active_jobs` when jobs are bulk enqueued. If the lambda is defined but returns `nil`, Solid Queue will use the adapter defined for the job.
660+
661+
### Enqueueing jobs in different shards
662+
663+
Individual jobs can be assigned to shards by leveraging their `queue_adapter` property:
664+
665+
```ruby
666+
class SomeJob < ApplicationJob
667+
self.queue_adapter = ActiveJob::QueueAdapters::SolidQueueAdapter.new(db_shard: :queue_shard_two)
668+
```
669+
670+
This job will be enqueued in the shard named `queue_shard_two`.
671+
672+
Alternatively you can define a lambda to implement a custom strategy for defining which shard a job will be enqueued to:
673+
674+
```ruby
675+
config.solid_queue.shard_selection_lambda = ->(active_job:, active_jobs:) { SolidQueue.connects_to[:shards].keys.sample } # pick a shard randomly
676+
```
677+
678+
612679
## Inspiration
613680

614681
Solid Queue has been inspired by [resque](https://github.com/resque/resque) and [GoodJob](https://github.com/bensheldon/good_job). We recommend checking out these projects as they're great examples from which we've learnt a lot.

0 commit comments

Comments
 (0)