You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: README.md
+67
Original file line number
Diff line number
Diff line change
@@ -609,6 +609,73 @@ my_periodic_resque_job:
609
609
610
610
and the job will be enqueued via `perform_later` so it'll run in Resque. However, in this case we won't track any `solid_queue_recurring_execution` record for it and there won't be any guarantees that the job is enqueued only once each time.
611
611
612
+
## Multisharding
613
+
614
+
If your application reaches a point where the pressure on the database used for jobs is such that you need to spread the load over multiple databases, then this section is for you.
615
+
616
+
You can extend the Solid Queue database configuration to use different shards:
617
+
618
+
```ruby
619
+
config.solid_queue.connects_to = {
620
+
shards: {
621
+
queue_shard_one: { writing: :queue_shard_one },
622
+
queue_shard_two: { writing: :queue_shard_two }
623
+
}
624
+
}
625
+
```
626
+
627
+
Queue database shards will need to have been defined in `config/database.yml` as shown in the installation section. Both shards need to share the same schema, and down the line share the same migration configuration:
Simply converting a simpler database configuration such as `config.solid_queue.connects_to = { database: { writing: :queue } }` to `config.solid_queue.connects_to = { shards: { queue: { writing: :queue } } }` will not have any effects on the behavior of Solid Queue.
645
+
646
+
### Configuration
647
+
648
+
In `config/environments/production.rb` or for the environment you want to enable Solid Queue in, you can define the following options:
- `config.solid_queue.primary_shard`is the shard that will be used to enqueue or schedule jobs without any specific adapter configuration. It defaults to the first shard found in `config.solid_queue.connects_to` (ActiveRecord default)
657
+
- `config.solid_queue.active_shard`is the shard that will be used by workers, dispatchers and schedulers to manage and process jobs. It defaults to the `primary_shard`.
658
+
With a basic Solid Queue configuration and the option described above you can start a worker and dispatcher working on the `queue_shard_two` shard by running `SOLID_QUEUE_ACTIVE_SHARD=queue_shard_two bin/jobs start`
659
+
- `config.solid_queue.shard_selection_lambda`helps you define a custom strategy to determine in which shard a job should be enqueued. It accepts keyword parameters `active_job` when a single job is enqueued or scheduled and `active_jobs` when jobs are bulk enqueued. If the lambda is defined but returns `nil`, Solid Queue will use the adapter defined for the job.
660
+
661
+
### Enqueueing jobs in different shards
662
+
663
+
Individual jobs can be assigned to shards by leveraging their `queue_adapter` property:
Solid Queue has been inspired by [resque](https://github.com/resque/resque) and [GoodJob](https://github.com/bensheldon/good_job). We recommend checking out these projects as they're great examples from which we've learnt a lot.
0 commit comments