Prevent enqueuing duplicate jobs in the first place ?

[jwoodrow]

Hi (again),

Would it be possible to prevent a job from ever being enqueued with a concurrency option (that would similarly to the one on the perform concurrency options) ?

I was thinking something along the lines of:

1. Add a new column global_uniqueness_key to SolidQueue::Job with an index and a unique constraint
2. add a rescue to fail silently on ActiveRecord::RecordNotUnique in create_from_active_job
3. If the necessary requirements in the job are set so a uniqueness key can be obtained, then add it to the attributes_from_active_job otherwise it's business as usual

The reason I think this could be useful is that enqueuing a job is a write task, which does take some time to run on the database, but mostly because when using enqueue_after_transaction_commit in combination with some after_create/update callbacks that might end up trying to enqueue the same job multiple times because of some sort of association callback saves funny business I think it would be preferable for some jobs to simply enqueue only once and drop the other enqueues (they would have been enqueued at the same time after all).

I think there would be one limitation though (unless you want to add dependencies or rely on Rails.cache to use it as a mutex or maybe theres a fully database integrated option I don't know) the limit to the amount of enqueued jobs with this uniqueness key could only be statically set to 1 because of the unique index.

Would be open to try my hand at it on my free time if it could be something interesting to add to SolidQueue

🙌

[rosa]

Hey @jwoodrow, thanks for taking the time to write this up! Would this be covered by #176?

[jwoodrow]

Hi @rosa, it's similar in the idea, but the main difference is I was more thinking of preventing duplicate jobs from being enqueued in the first place.

let's say we have this, admitidly very bad, use case

ApplicationRecord.enqueue_after_transaction_commit = true

class SomeModel < ApplicationRecord
  belongs_to :some_other_model, optional: true

  after_create :enqueue_create_elastic_search_job
  after_update :enqueue_update_elastic_search_job
  after_update :some_badly_thought_method

  def enqueue_create_elastic_search_job
    CreateElasticeSearchJob.perform_later(id)
  end

  def enqueue_update_elastic_search_job
    UpdateElasticSearchJob.perform_later(id)
  end

  def some_badly_thought_method
    create_some_other_model!
  end
end

def SomeOtherModel < ApplicationRecord
  has_one :some_model

  after_save :some_updating_method

  def some_updating_method
    return unless some_condition?

    some_model.update!(some_attribute: some_value)
  end
end

some_model = SomeModel.create

some_model.update(an_attribute: 'a_value')

with #176
CreateElasticeSearchJob would be enqueued once and UpdateElasticSearchJob would be enqueued twice but executed only once

with my proposal
CreateElasticeSearchJob would be enqueued once and UpdateElasticSearchJob would be enqueued once and would fail to be inserted the second time meaning it wouldn't even need to be discarded at a later time

Hope I'm explaining things right 😅 I noticed issues of duplicate jobs being enqueued in our app when activating enqueue_after_transaction_commit and realized it came from spagetti code that was monkey patched by us into resque to check if a job was already in a queue before attempting to enqueue another one (always specifying keys for what is considered "unique") by using redis functions basically

[rosa]

the main difference is I was more thinking of preventing duplicate jobs from being enqueued in the first place.

But the concurrency controls take effect on enqueuing time 🤔 Right now, the job is blocked if there's a conflicting job in the queue (or being run). That change is about doing the same but discarding the job instead. Or do you mean you would want to prevent the job from being enqueued forever, even when the previous job has already been run and everything? Or even when it's scheduled to be run in the future?

[jwoodrow]

Maybe I'm the one who misunderstood, I thought concurrency was only there to prevent execution of jobs not to prevent jobs from being queued up in the first place because in the Readme there are mentions of "running"

Solid Queue extends Active Job with concurrency controls, that allows you to limit how many jobs of a certain type or with certain arguments can run at the same time. When limited in this way, jobs will be blocked from running, and they'll stay blocked until another job finishes and unblocks them, or after the set expiry time (concurrency limit's duration) elapses. Jobs are never discarded or lost, only blocked.

But if the concurrency controls take place when enqueuing then having a discarding option at that part would in fact work as I wanted it to

[rosa]

Yes, that's right. It's explained in the concurrency section, a bit below that introductory paragraph.

[jwoodrow]

Ah well I'll close this since it's a duplicate then, thanks !