generic topic to pool sync

[mccanne]

Zinger should have options to run more generically by syncing a kafka topic to a pool without the strict enforcement of the sequential offset field in the kafka meta record. Here we would just sync the kafka key/value or just the value without creating a meta record. Also you should be able to run with auto-commit of consumer commit offsets so multiple processes could sync in parallel to the same data lake (where strict ordering does not matter).

The auto-commit approach creates a window where data could be dropped from the lake and committed. We should think through how we might use explicit commits that would be performed after the pool commit with some way of recovering from a crash if the pool is committed to but the topic commit offset is not updated. Maybe we need a kafka meta-field after all to make this work.