Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] [Postgres] Resumable replication per-table #163

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

rkistner
Copy link
Contributor

@rkistner rkistner commented Dec 11, 2024

Some history:
#150 added support to resume initial replication on a per-table level (skipping tables already replicated).
#152 improved table snapshots to be faster when retrying. However, with a 5-minute snapshot timeout, we're still limited to around 2.5 million rows per table.
#160 helped to increase the number of rows slightly, to around 3 million rows per table. This is likely due to results being buffered before the query times out.

This attempts to chunk initial snapshot queries using the primary key, allowing us to resume initial replication, extending the number of rows we can replicate significantly. Right now, this is limited to tables with a single primary key column.

TODO:

  • Persist last seen key, allowing us to actually resume the replication.
  • Test with every possible key type, or limit supported types.
  • Test edge cases with TOAST values.

Copy link

changeset-bot bot commented Dec 11, 2024

⚠️ No Changeset found

Latest commit: f1ff6db

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Base automatically changed from fix-slot-recovery to main December 12, 2024 08:51
@rkistner rkistner force-pushed the resumable-replication-2 branch from fd8ade8 to 3cb3248 Compare December 12, 2024 12:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant