You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
These changes are part of checkpoint write support.
To build a V1 checkpoint, we need to:
Perform Log Segment Replay to retrieve all action batches for the table's state
Each batch has to be scanned and filtered to only include actions to be written to the V1 checkpoint file.
The FileActionsVisitor and NonFileActionsVisitor visitors need to be applied to each batch.
Introduce V1CheckpointLogReplayScanner, a component responsible for filtering actions during log replay to include only those necessary for constructing a V1 checkpoint with the new visitors.
Describe the functionality you are proposing.
This scanner should:
Retain only the most recent protocol and metadata actions.
Deduplicate transaction actions per app ID.
Remove duplicate file actions based on path and unique ID.
Exclude tombstones older than minimum_file_retention_timestamp.
Additionally, introduce v1_checkpoint_actions_iter to leverage this scanner when iterating through log actions.
Additional context
No response
The text was updated successfully, but these errors were encountered:
Please describe why this is necessary.
These changes are part of checkpoint write support.
To build a V1 checkpoint, we need to:
The
FileActionsVisitor
andNonFileActionsVisitor
visitors need to be applied to each batch.Introduce
V1CheckpointLogReplayScanner
, a component responsible for filtering actions during log replay to include only those necessary for constructing a V1 checkpoint with the new visitors.Describe the functionality you are proposing.
This scanner should:
Retain only the most recent protocol and metadata actions.
Deduplicate transaction actions per app ID.
Remove duplicate file actions based on path and unique ID.
Exclude tombstones older than minimum_file_retention_timestamp.
Additionally, introduce
v1_checkpoint_actions_iter
to leverage this scanner when iterating through log actions.Additional context
No response
The text was updated successfully, but these errors were encountered: