Difficulty starting continuous replication with child process

[amkurtis]

I have noted stale data in the app I have set up, with long lag periods. I'm running a Go app using v0.3.7 with the subprocess -exec command on Fly.io, and streaming to Backblaze.

Do I need to modify the configuration for continuous sync?

My impression is that changes to the database are not being automatically streamed, and that I have not successfully started the replicate command. My second impression is I've misunderstood how it updates, but anticipated essentially syncs on change with up to 10s (by default) of delay.

For example, this evening: if I run litestream generations snapshots, or wal, I saw data with only one older generation listed:

• I have added and can add new data since this date which does not appear to sync.
• On restore, the database contained the data present in the generation listed above but no data added since then (as expected given the generation end-time, but new data was present in the live database)
• If I add new data and SSH in and run litestream generations/snapshots/wal, I can see it has updated in the container as the end-time has updated. But this does not appear to be replicated to S3
• Litestream does correctly pull the database when the container is first started with the latest generation present in S3

I can manually force a sync by:

• SSHing into the container and running litestream replicate, or
• Closing + restarting the container, but this does make me hesitant in case it does not sync prior to shutdown

Litestream generations/snapshots/wal then show me a more up-to-date copy, at the time the container shut-down or when I SSH-ed in to run the command.

The log messages I can find around replication/restoration are on start only:

My setup is inspired by the example at https://github.com/benbjohnson/litestream-docker-example, using a static binary.

run.sh

`#!/bin/bash
set -e

# Restore the database if it does not already exist.
if [ -f /data/db ]; then
	echo "Database already exists, skipping restore"
else
	echo "No database found, restoring from replica if exists"
	litestream restore -v -if-replica-exists -o /data/db "${REPLICA_URL}"
fi

# Run litestream with your app as the subprocess.
exec litestream replicate -exec "/usr/local/bin/myapp -dsn /data/db"`

litestream.yml

dbs:
  - path: /data/db
    replicas:
      - url: s3://{{x}}.us-west-000.backblazeb2.com/db
        retention: 720h

Thank you!

[benbjohnson]

@kurtis-poole Sorry for the delay getting back to you. I got my booster shot on Wednesday and it knocked me out pretty good.

Hmm, that is an odd behavior. It's strange that Litestream picks up the snapshot but nothing else. A couple questions to help debug:

1. Is your application updating the journal mode? Litestream sets the database to WAL mode but I could see it breaking if you're reseting journal_mode to something else.

2. Do you see a file called /data/db-wal and is it growing as you're making transactions against your database?

3. Can you run a tree command against the /data/.db-litestream directory and see what it shows? I'm curious if Litestream is copying out the WAL pages but not replicating them out.

4. Do you see any files under /db/generations/f532bb5b97c633bc/wal on Backblaze?

[amkurtis]

No apology for delay necessary, grateful for help

1. Yes it does, but to WAL:

func Open(dsn string) (*sql.DB, error) {
	db, err := sql.Open("sqlite", dsn)
	if err != nil {
		return nil, err
	}

	if _, err := db.Exec(`PRAGMA journal_mode = wal;`); err != nil {
		return nil, fmt.Errorf("enable wal: %w", err)
	}

	return db, nil
}

2. /data/db-wal exists, size grows if I create a new record:

3. tree .db-litestream output:

4. Yep, 213 files for /db/generations/f532bb5b97c633bc/wal with the latest date modified at 2021-12-30 05:59 which matches the generation end

Some more digging on 4 with the most recent generation, and does appear Litestream is sending changes upstream. Checked to re-confirm state on the container compared to remotely on my machine to see whether any change and output as below, still appears desynced:

Container with app

Remote

But the files in gen 98c-xx do appear to be up to date in the Backblaze bucket:

This suggests to me Litestream is syncing changes to the bucket as intended. But this does not appear recoverable without forcing a 'sync', and perhaps the 'sync' is simply forcing a generation change that then gets picked up?

[benbjohnson]

Hmm, that's odd that Litestream is not seeing the WAL files on Backblaze. Would it be possible for you to try it against an AWS S3 bucket? All the S3-compatible replica types use the same code so I'm trying to narrow down if it's an issue with the B2 API specifically.

[amkurtis]

Works immediately on AWS! Thanks. New data syncs, restore gives me new data and the lag is as expected (ie. < one second), so does seem like a B2 thing. I'll swap over to a different S3 service.

An unrelated error to B2: but adding aws has resulted in a no snapshot available error, which logs on init and then every second once any change to the restored database occurs. I removed the B2 bucket so that restore would occur from AWS, and this occurs as expected, but this error persists:

[benbjohnson]

Awesome! I'm glad S3 works. You may need to stop Litestream, delete the /data/.db-litestream directory, and restart Litestream to get rid of the snapshot error. Litestream doesn't currently handle switching replicas in the config very cleanly. I'll need to get that fixed up.