Skip to content

Commit 57bd18b

Browse files
rkistnerstevensJourneyRentacookiedylanvorster
authored
Module Replication: MongoDB & MySQL support (#133)
* added table debug info calls * Update Walstream classes to use the factories from the service context Simplified configuration of Replication module * added connection schema api method and initial workings for replication lag * update replication lag API. Cleanup queries * cleanup * cleanup retried queries. Add SSL config for MySQL config * Use framework errors. Update DataSourceConfig and ResolvedDataSourceConfig. Added DatabaseSchemaV2 type to remove Postgres fields. * more mysql cleanup * update failing Github action name to be clearer * share some utilities. Playing around with replication * unfortunate implementation of replication lag * track binlog positions in replicated GTID identifier * Introduces the AbstractReplicator concept as a more flexible alternative to the ReplicationAdapter * wip * Cleaned up WalStream Replicator and ReplicationJobs Fixed test imports * Removed no longer used Teardown function Moved Replication Module definition to the replication packaged * Fixed outdated comments and exporting only used Replication classes * Updated and re-enabled postgres tests * Updated lock file * more hacks to get writerows to work * wip: before implementing update * added zongji types * update typescript * shuffle files arround * update after shuffle * Test: Try changing order of filter expression in github test action * Replication job start no longer immediately cancels * use new replicator interfaces * Move connection factory shutdown Pass along publication name * move toSQLite function to common * update types * move * Made column type optional since for Postgres the column type is not propagated on WalStream schema updates. * Fixed (hopefully) the test filtering for the test github action * Corrected exclusion filter for test workflow * Added build:tests directive to postgres module * Export populate_test_data * Export the test utils from postgres module * Moved data generation method to test utils * Moved data generation utils to tests * Removed unused export * Updated test tsconfig to include jpgwire * Try with outDir specified * Import from dist * Revert file move and now using import from dist * cleanup * fix build error * Ensure collections are created for tests * Added dedicated BucketStorageFactory for Postgres tests Reverted auto collection creation for BucketStorageFactory in the service-core tests * Moved custom factory into the test it is being used at * Revert factory imports * Revert noEmit for postgres tests * Trying moved factory again after tsconfig fix * Create test collection conditionally * Added a few more TODOs Cleanup Updated comments and docs where appropriate * Some more cleanup Renamed StorageFactoryProvider to StorageEngine * Consolidated getDebugTableInfo Fixed route storage references post rename * Removed Postgres ZeroLSN from BucketStorage * Writecheckpoint fix after conflict * fix: align debug_api field for compatibility (#6) * Implement powersync instance teardown functionality Reworked replication config resource cleanup Streamlined storage provider and engine a bit. Exposed option to dispose of storage. Made some engines in the service context optional * add some helpers * Adjusted checks in routes for the the RouterEngine * Updated lockfile * Renamed storage provider storage dispose method. * Update lockfile * Basic MongoDB replication structure. * Support resuming; multi-document operations. * Workaround for transaction boundaries; filter ns in pipeline. * Proper initial snapshot + streaming afterwards. * Use _id directly as replica identity. * Configurable defaultSchema. * Use _id directly as replica identity. * Configurable defaultSchema. * Fix tests. * Use 'test_schema' instead of 'public' for tests. * Expand comment. * Tweaks for ReplicaId. * Fix subkey calculation & tests. * Fix more tests. * Fix merge conflicts. * Fix Dockerfile. * Fix and test mongo data type conversion. * More mongo format tests. * Fix initial snapshot; initial change-stream tests. * Fixed missing ErrorRateLimiter for WalStreamReplicationJob * Record keepalive events. * Another fix; basic replication tests passing. * Added more logging for compact action * Handle collection drop and rename events. * Handle replace event. * Handle missing fullDocument correctly. * Improve keepalive stability. * Added start up and shutdown logging to engines. Consolidated shutdown functions of engines. * Use checkpoints for standard replication. * Support wildcard table patterns. * Fix merge issue. * Added changeset * Made Postgres Module package public Removed mongodb test dependency in Postgres module * Updated lockfile * Update typescript, vitest, prettier. * Apply prettier changes. * Fix type issues. * Fix tests. * Add changeset. * Fix aborting logic. * Normalize mongo connection parameters. * Remove error message when closing a ChangStreamReplicationJob. * Support M0 clusters; better error message on unsupported clusters. * Fix sharded cluster check. * Fix initial replication regression. * Refactor schema definitions to not be postgres-specific. * Rename original_type -> internal_type. * Include source types in the generated schema comments (optional). * Explicit TableSchema type. * Add changeset. * remove mongodb publish config restriction in order to publish dev packages * need to explicitly set MongDB package access to public for publish. * Post-merge test fixes. * Fix pnpm-lock. * Implement basic diagnostics apis for MongoDB. * Add test for getConnectionSchema. * Improve schema filtering; return defaultSchema in API. * Initialize tag variable before use in nested function call. * Add comment for tag default value * Changeset * Comment update * list all databases (#96) * [MongoDB] Schema Endpoint Fix (#97) * test: populate schema type field * pg_type restore * Lockfile update and merge conflict fixes * Added dev docker compose for mysql * Added MySQL connection manager and use appropriate connections * Added stricter type definition for checkpoints Moved BinlogStream Added type mapper for mysql -> sqlite types Removed schemaV2 since it is no longer needed * Fixed diagnostics route merge conflict Renamed binlog replication job * Add MySQLConnection management Updated config for the Zongji binlog listener * Made streamable mysql connections available * Updated BinlogStream to use appropriate connections for snapshot streaming. Lots of cleanup and consolidation * Lockfile update * [Modules] Feat: Replication Events (#105) * Ignore auth errors on collections for mongodb schema. * Add changeset. * Filter out views. * Updated dev/test mysql docker compose Renamed MysqlBinLogStream Some naming cleanup Fixed connection usage in BinlogStream * Added BinlogStream tests (WiP) * Initializing batch in constructor of MongoBucketBatch * Some MongoModule merge conflict handling and cleanup * Made mysql module publishing public * Updated dockerfile * Added mysql module to service tsconfig * Fixed dockerfile mysql module copy * Updated zongji dev package version Some cleanup of modules package.json * updates after pulling in main branch * Updated vitest * Added configuration error handling Filter out tables not in sync rules Improve abort logic handling * Don't start replication if already aborted * Exposed DateStrings connection option in Zongji constructor Using updated binlog listener package * Lockfile * Added ConnectionTester interface Replication Modules now implement this interface. Removed singleton module exports * [Modules] Move Write Checkpoint APIs (#110) * Made it possible to specify timezone on zongji listener configuration to stop unwanted timezone skew Added serverId configuration Added check for binlog existence before starting replication. * Correctly handle date parsing on binlog events and table snapshotting * Some cleanup * fix esm stuff * Using syncrule id for MySQL serverId * Changeset * Prevent mysql connection manager throwing errors on shutdown. * Improved shutdown logic of binlog stream * Renamed Checkpoint to ReplicationCheckpoint * Reduce required permissions for replicating from a single database. * Add changeset. * fix: Cached Parsed Sync Rules (#114) * MySQL Type Consolidation (#115) MySQL: - Added generator function for MySQL serverId - Logging cleanup - Using JsonContainer for Json values in MySQL - Unified type mappings for replicated and snapshot mysql data types - Added test running config to github actions - Mapping all integer types to bigint - Handling Set data types as json array - Added test for float mapping * Properly replicate additional MongoDB types: Decimal, RegExp, MinKey, MaxKey. * Test mongodb on GA. * Also test with MongoDB 8.0. * Support compacting specific buckets when compacting manually. * Add configuration for using mongodb postimages. * Increase timeout between writes in tests. * Use "await using" to simplify tests. * Add postImage tests. * Test and fix wildcard collections. * Invalidate changestream if postImage is not available. * Rename post_images config option. * Parse returned mysql schema result (#122) * Bugfix:Parse returned schema info as an object. Previously this was the default, but since a recent change to the mysql connection options, all JSON fields are now returned as strings. * More MongoDB type fixes and tests. * Handle improperly formatted mysql version strings (#124) Handled MySQL version checks better and enabled tests for MySQL 5.7 * Automatically clear errors when restarting replication. * Use commit instead of keepalive. * Fix initial snapshot implementation. * Avoid collMod permission on _powersync_checkpoints. * Minor cleanup. * Validate changeStreamPreAndPostImages on existing collections. * Avoid current_data document storage for MongoDB. * Fix tests. * Use $changeStreamSplitLargeEvent to handle large updates. (#130) * Fix tests. * Fix managed write checkpoint filtering. (#134) * Support json_each in parameter queries (#126) * Proof-of-concept: json_each support in parameter queries. * Extract out json_each. * Fixes. * Tweak usesDangerousRequestParameters check. * Fix handling of nested json. * Fix tableName regression. * Add changeset. --------- Co-authored-by: stevensJourney <[email protected]> * Modular Architecture Prep for Merge (#137) * remove postgres errors from mysql+mongodb rate limiters * use common typescript version * update changeset * update zongji package --------- Co-authored-by: Steven Ontong <[email protected]> Co-authored-by: Roland Teichert <[email protected]> Co-authored-by: stevensJourney <[email protected]> Co-authored-by: Roland Teichert <[email protected]> Co-authored-by: Dylan Vorster <[email protected]>
1 parent 1c99a4d commit 57bd18b

File tree

282 files changed

+15010
-5755
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

282 files changed

+15010
-5755
lines changed

Diff for: ‎.changeset/cold-items-explain.md

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@powersync/service-module-mongodb': patch
3+
---
4+
5+
Fix diagnostics schema authorization issues for MongoDB

Diff for: ‎.changeset/fifty-dogs-reply.md

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@powersync/service-module-mongodb': minor
3+
---
4+
5+
Reduce permissions required for replicating a single mongodb database

Diff for: ‎.changeset/gentle-icons-try.md

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@powersync/service-module-mysql': patch
3+
---
4+
5+
Fixed MySQL version checking to better handle non-semantic version strings

Diff for: ‎.changeset/green-peas-roll.md

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
'@powersync/service-module-mongodb': minor
3+
'@powersync/service-image': minor
4+
---
5+
6+
Add MongoDB support (Alpha)

Diff for: ‎.changeset/healthy-rules-arrive.md

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@powersync/service-module-mysql': patch
3+
---
4+
5+
Fixed mysql schema json parsing

Diff for: ‎.changeset/heavy-shirts-chew.md

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@powersync/service-core': patch
3+
---
4+
5+
Improved sync rules storage cached parsed sync rules, accommodating different parsing options where necessary.

Diff for: ‎.changeset/lemon-terms-play.md

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
'@powersync/service-module-mysql': minor
3+
---
4+
5+
Generate random serverId based on syncrule id for MySQL replication client
6+
Consolidated type mappings between snapshot and replicated values
7+
Enabled MySQL tests in CI
8+

Diff for: ‎.changeset/olive-spoons-stare.md

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@powersync/service-core': patch
3+
---
4+
5+
Moved tag variable initialization in diagnostics route to ensure it is initialized before usage

Diff for: ‎.changeset/orange-eagles-tap.md

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@powersync/lib-services-framework': minor
3+
---
4+
5+
Added disposable listeners and observers

Diff for: ‎.changeset/popular-snails-cough.md

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
'@powersync/service-core': minor
3+
'@powersync/service-sync-rules': minor
4+
---
5+
6+
Added ability to emit data replication events

Diff for: ‎.changeset/rotten-pumas-protect.md

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
---
2+
'@powersync/service-core': minor
3+
'@powersync/service-module-mysql': minor
4+
'@powersync/service-sync-rules': minor
5+
---
6+
7+
Introduced alpha support for MySQL as a datasource for replication.
8+
Bunch of cleanup
9+

Diff for: ‎.changeset/slow-stingrays-kiss.md

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@powersync/service-core': minor
3+
---
4+
5+
Moved Write Checkpoint APIs to SyncBucketStorage

Diff for: ‎.changeset/sour-turkeys-collect.md

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
'@powersync/service-module-postgres': patch
3+
'@powersync/service-rsocket-router': patch
4+
'@powersync/service-types': patch
5+
---
6+
7+
Updates from Replication events changes

Diff for: ‎.changeset/tender-vans-impress.md

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
---
2+
'@powersync/service-core': minor
3+
'@powersync/service-sync-rules': minor
4+
'@powersync/lib-services-framework': minor
5+
'@powersync/service-jpgwire': minor
6+
'@powersync/service-types': minor
7+
'@powersync/service-image': major
8+
'@powersync/service-module-postgres': patch
9+
---
10+
11+
- Introduced modules to the powersync service architecture
12+
- Core functionality has been moved to "engine" classes. Modules can register additional functionality with these engines.
13+
- The sync API functionality used by the routes has been abstracted to an interface. API routes are now managed by the RouterEngine.
14+
- Replication is managed by the ReplicationEngine and new replication data sources can be registered to the engine by modules.
15+
- Refactored existing Postgres replication as a module.
16+
- Removed Postgres specific code from the core service packages.

Diff for: ‎.changeset/violet-garlics-know.md

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@powersync/service-sync-rules': minor
3+
---
4+
5+
Support json_each as a table-valued function.

Diff for: ‎.changeset/weak-cats-hug.md

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@powersync/service-sync-rules': minor
3+
---
4+
5+
Optionally include original types in generated schemas as a comment.

Diff for: ‎.github/workflows/test.yml

+171-7
Original file line numberDiff line numberDiff line change
@@ -22,22 +22,67 @@ jobs:
2222
- name: Set up Docker Buildx
2323
uses: docker/setup-buildx-action@v3
2424

25-
- name: Build and push
25+
- name: Test Build Docker Image
2626
uses: docker/build-push-action@v5
2727
with:
2828
cache-from: type=registry,ref=stevenontong/${{vars.DOCKER_REGISTRY}}:cache
2929
context: .
3030
platforms: linux/amd64
3131
push: false
3232
file: ./service/Dockerfile
33-
# TODO remove this when removing Journey Micro
34-
build-args: |
35-
GITHUB_TOKEN=${{secrets.RESTRICTED_PACKAGES_TOKEN}}
3633

37-
run-tests:
38-
name: Test
34+
run-core-tests:
35+
name: Core Test
3936
runs-on: ubuntu-latest
4037

38+
steps:
39+
- uses: actions/checkout@v4
40+
41+
- name: Start MongoDB
42+
uses: supercharge/[email protected]
43+
with:
44+
mongodb-version: '6.0'
45+
mongodb-replica-set: test-rs
46+
47+
- name: Setup Node.js
48+
uses: actions/setup-node@v4
49+
with:
50+
node-version-file: '.nvmrc'
51+
52+
- uses: pnpm/action-setup@v4
53+
name: Install pnpm
54+
with:
55+
version: 9
56+
run_install: false
57+
58+
- name: Get pnpm store directory
59+
shell: bash
60+
run: |
61+
echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_ENV
62+
63+
- uses: actions/cache@v3
64+
name: Setup pnpm cache
65+
with:
66+
path: ${{ env.STORE_PATH }}
67+
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
68+
restore-keys: |
69+
${{ runner.os }}-pnpm-store-
70+
71+
- name: Install dependencies
72+
run: pnpm install
73+
74+
- name: Build
75+
shell: bash
76+
run: pnpm build
77+
78+
- name: Test
79+
run: pnpm test --filter '!./modules/*'
80+
81+
run-postgres-tests:
82+
name: Postgres Test
83+
runs-on: ubuntu-latest
84+
needs: run-core-tests
85+
4186
strategy:
4287
fail-fast: false
4388
matrix:
@@ -97,4 +142,123 @@ jobs:
97142
run: pnpm build
98143

99144
- name: Test
100-
run: pnpm test
145+
run: pnpm test --filter='./modules/module-postgres'
146+
147+
run-mysql-tests:
148+
name: MySQL Test
149+
runs-on: ubuntu-latest
150+
needs: run-core-tests
151+
152+
strategy:
153+
fail-fast: false
154+
matrix:
155+
mysql-version: [5.7, 8.0, 8.4]
156+
157+
steps:
158+
- uses: actions/checkout@v4
159+
160+
- name: Start MySQL
161+
run: |
162+
docker run \
163+
--name MySQLTestDatabase \
164+
-e MYSQL_ROOT_PASSWORD=mypassword \
165+
-e MYSQL_DATABASE=mydatabase \
166+
-p 3306:3306 \
167+
-d mysql:${{ matrix.mysql-version }} \
168+
--log-bin=/var/lib/mysql/mysql-bin.log \
169+
--gtid_mode=ON \
170+
--enforce_gtid_consistency=ON \
171+
--server-id=1
172+
173+
- name: Start MongoDB
174+
uses: supercharge/[email protected]
175+
with:
176+
mongodb-version: '6.0'
177+
mongodb-replica-set: test-rs
178+
179+
- name: Setup NodeJS
180+
uses: actions/setup-node@v4
181+
with:
182+
node-version-file: '.nvmrc'
183+
184+
- uses: pnpm/action-setup@v4
185+
name: Install pnpm
186+
with:
187+
version: 9
188+
run_install: false
189+
190+
- name: Get pnpm store directory
191+
shell: bash
192+
run: |
193+
echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_ENV
194+
195+
- uses: actions/cache@v3
196+
name: Setup pnpm cache
197+
with:
198+
path: ${{ env.STORE_PATH }}
199+
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
200+
restore-keys: |
201+
${{ runner.os }}-pnpm-store-
202+
203+
- name: Install dependencies
204+
run: pnpm install
205+
206+
- name: Build
207+
shell: bash
208+
run: pnpm build
209+
210+
- name: Test
211+
run: pnpm test --filter='./modules/module-mysql'
212+
213+
run-mongodb-tests:
214+
name: MongoDB Test
215+
runs-on: ubuntu-latest
216+
needs: run-core-tests
217+
218+
strategy:
219+
fail-fast: false
220+
matrix:
221+
mongodb-version: ['6.0', '7.0', '8.0']
222+
223+
steps:
224+
- uses: actions/checkout@v4
225+
226+
- name: Start MongoDB
227+
uses: supercharge/[email protected]
228+
with:
229+
mongodb-version: ${{ matrix.mongodb-version }}
230+
mongodb-replica-set: test-rs
231+
232+
- name: Setup Node.js
233+
uses: actions/setup-node@v4
234+
with:
235+
node-version-file: '.nvmrc'
236+
237+
- uses: pnpm/action-setup@v4
238+
name: Install pnpm
239+
with:
240+
version: 9
241+
run_install: false
242+
243+
- name: Get pnpm store directory
244+
shell: bash
245+
run: |
246+
echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_ENV
247+
248+
- uses: actions/cache@v3
249+
name: Setup pnpm cache
250+
with:
251+
path: ${{ env.STORE_PATH }}
252+
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
253+
restore-keys: |
254+
${{ runner.os }}-pnpm-store-
255+
256+
- name: Install dependencies
257+
run: pnpm install
258+
259+
- name: Build
260+
shell: bash
261+
run: pnpm build
262+
263+
- name: Test
264+
run: pnpm test --filter='./modules/module-mongodb'

Diff for: ‎.prettierignore

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
**/.git
2+
**/node_modules
3+
dist
4+
lib
5+
pnpm-lock.yaml

Diff for: ‎README.md

+4-3
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
<a href="https://www.powersync.com" target="_blank"><img src="https://github.com/powersync-ja/.github/assets/7372448/d2538c43-c1a0-4c47-9a76-41462dba484f"/></a>
33
</p>
44

5-
*[PowerSync](https://www.powersync.com) is a sync engine for building local-first apps with instantly-responsive UI/UX and simplified state transfer. Syncs between SQLite on the client-side and Postgres, MongoDB or MySQL on the server-side.*
5+
_[PowerSync](https://www.powersync.com) is a sync engine for building local-first apps with instantly-responsive UI/UX and simplified state transfer. Syncs between SQLite on the client-side and Postgres, MongoDB or MySQL on the server-side._
66

77
# PowerSync Service
88

@@ -11,6 +11,7 @@
1111
The service can be started using the public Docker image. See the image [notes](./service/README.md)
1212

1313
# Monorepo Structure:
14+
1415
## Packages
1516

1617
- [packages/service-core](./packages/service-core/README.md)
@@ -52,13 +53,13 @@ Contains the PowerSync Service code. This project is used to build the `journeya
5253

5354
- [docs](./docs/README.md)
5455

55-
Technical documentation regarding the implementation of PowerSync.
56+
Technical documentation regarding the implementation of PowerSync.
5657

5758
## Test Client
5859

5960
- [test-client](./test-client/README.md)
6061

61-
Contains a minimal client demonstrating direct usage of the HTTP stream sync API. This can be used to test sync rules in contexts such as automated testing.
62+
Contains a minimal client demonstrating direct usage of the HTTP stream sync API. This can be used to test sync rules in contexts such as automated testing.
6263

6364
# Developing
6465

Diff for: ‎libs/lib-services/package.json

+3-1
Original file line numberDiff line numberDiff line change
@@ -26,11 +26,13 @@
2626
"dotenv": "^16.4.5",
2727
"lodash": "^4.17.21",
2828
"ts-codec": "^1.2.2",
29+
"uuid": "^9.0.1",
2930
"winston": "^3.13.0",
3031
"zod": "^3.23.8"
3132
},
3233
"devDependencies": {
3334
"@types/lodash": "^4.17.5",
34-
"vitest": "^0.34.6"
35+
"@types/uuid": "^9.0.4",
36+
"vitest": "^2.1.1"
3537
}
3638
}

0 commit comments

Comments
 (0)