Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Discard duplicate jobs #176

Open
matteeyah opened this issue Mar 12, 2024 · 14 comments · May be fixed by #523
Open

Discard duplicate jobs #176

matteeyah opened this issue Mar 12, 2024 · 14 comments · May be fixed by #523
Assignees

Comments

@matteeyah
Copy link

Summary

The concurrency controls that currently exist allow blocking execution of "duplicate" jobs, or jobs with the same arguments. These jobs wait, then execute after the currently running job finishes. There's no way to discard all duplicate jobs and prevent them from running completely.

Proposal

Add a way to discard jobs that get blocked by the concurrency controls.

@rosa
Copy link
Member

rosa commented Mar 13, 2024

@matteeyah, yes! I've got this on my mind as well! In the end, I'm not sure there's a version of #105 that I like or that it's useful for what we need, so I thought, in the meantime, to add another parameter to limits_concurrency to discard instead of blocking. Something like:

on_conflict: :discard

By default, it'd be:

on_conflict: :block

which would be the current behaviour.

@matteeyah
Copy link
Author

That's exactly what I had in mind!


For reference - We're currently using https://github.com/veeqo/activejob-uniqueness. It serves a slightly different purpose, but it also checks for duplicate jobs.

It also has an on_conflict option - it supports logging a conflict, raising an error on a conflict and passing a proc to handle the conflict manually.

@nhorton
Copy link

nhorton commented Apr 16, 2024

Another vote for following the activejob-uniqueness conventions.

I would note that of the strategies offered by the gem, until_and_while_executing is the one we have found most-useful.

@nhorton
Copy link

nhorton commented May 17, 2024

Is there any more word on this? Or any openness to others trying to contribute?

@rosa rosa added this to the v1.0 milestone Jul 18, 2024
@rosa
Copy link
Member

rosa commented Jul 18, 2024

Hey @nhorton, sorry for the delay! I'm back at this now and would like to get this feature ready for version 1.0.

of the strategies offered by the gem, until_and_while_executing is the one we have found most-useful.

I think that would be the only strategy supported because of the way concurrency controls work. Right now jobs don't unblock other jobs or release the semaphore until they complete, and for simplicity, it'd be much easier to keep it that way.

Or any openness to others trying to contribute?

Definitely! 🙌

@bilby91
Copy link

bilby91 commented Jul 19, 2024

@rosa Is there an ETA for 1.0 ?

@rosa
Copy link
Member

rosa commented Jul 19, 2024

@bilby91, my hope is by the end of August but you know, there are always unexpected things 😅 😅

@bilby91
Copy link

bilby91 commented Jul 22, 2024

@rosa Awesome! I look forward to testing this feature.

@rosa
Copy link
Member

rosa commented Sep 13, 2024

In the end I won't be able to get this one in for v1.0, but I hope to get it there shortly after. Sorry!

@rosa rosa removed this from the v1.0 milestone Sep 13, 2024
@rosa rosa marked this as a duplicate of #105 Feb 5, 2025
@wheee
Copy link

wheee commented Feb 7, 2025

Hey @rosa , any chance that this will come about sometime in the near/medium future? Would love to use this feature as well!

@rosa
Copy link
Member

rosa commented Feb 7, 2025

I'm quite tight on time at the moment, but if someone wants to submit a PR, I'll be happy to review!

@joelzwarrington
Copy link

Hi @rosa 👋

I took a stab at this in #523 if you wouldn't mind taking a look 😊

@joelzwarrington
Copy link

Hey @rosa! Any chance you can have a look at #523 sometime soon? Thanks 🙇

@rosa
Copy link
Member

rosa commented Mar 5, 2025

@joelzwarrington yes, sorry! I was out at a conference since Wednesday last week and came back yesterday, and then had to catch up with a bunch of other stuff at work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants