Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch SQL: Support Cron syntax for polling schedule #1949

Open
willdonnelly opened this issue Sep 16, 2024 · 0 comments
Open

Batch SQL: Support Cron syntax for polling schedule #1949

willdonnelly opened this issue Sep 16, 2024 · 0 comments

Comments

@willdonnelly
Copy link
Member

willdonnelly commented Sep 16, 2024

Currently our batch SQL connectors support two forms of schedule descriptions: Go duration strings like "5h" or "30m" or a special once-per-day schedule like "daily at 11:23Z". There are plenty of other schedules users might desire which cannot currently be expressed, mostly weekly, monthly, or bimonthly patterns.

If we supported cron expressions we could handle all of those and most other patterns people are likely to want, for instance:

  • Once a week at 1:23AM on Thursdays: 1 23 * * 4
  • Once a month at 5:00AM on the 1st: 0 5 1 * *
  • Twice a month at 5:00Am on the 1st and 15th: 0 5 1,15 * *

The actual work here is basically:

  1. Finding and vetting an appropriate cron expressions package which we can use.
  2. Deciding how to represent cron expressions as a freeform string input in the schedule property, for instance "cron(1 23 * * 4)" or "cron when 1 23 * * 4" or something.
  3. Detecting that pattern in schedule.Parse() and delegating to the cron expressions package.
  4. Adding cron expressions to the schedule validation regex in each batch connector.
@willdonnelly willdonnelly changed the title Support Cron syntax for DB batch connector polling Batch SQL: Support Cron syntax for polling schedule Sep 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant