Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
source-zendesk-support: checkpoint during
AuditLogs
stream
For tasks that have a significant number of audit logs in Zendesk Support, it can take more than 24 hours to backfill. With the current `AuditLogs` stream strategy of reading records in descending order then stopping when we've reached a record we already read, we can't checkpoint until the backfill is completed. This means tasks that take more than 24 hours to backfill `AuditLogs` can get stuck restarting & attempting to backfill. This PR updates the `AuditLogs` stream to read records in ascending order. This lets the connector checkpoint after each response & pick up where it left off whenever it's restarted. Some noteable changes include: - Passing a list containing `start_time` and `end_time` for the `filter[created_at][]` query param to bound the responses' results to the specified timespan. - Pushing `end_time` 30 seconds in the past to avoid missing records if we query Zendesk in the middle of creating multiple records with the same `created_at`. - Keeping `AuditLogs` on the older `get_updated_state` method of updating state instead of migrating to the newer `state` property. This is to avoid rebackfilling all existing tasks.
- Loading branch information