Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[server][dvc] transfer offsetRecord and storeVersionState during blob transfer #1187

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

jingy-li
Copy link
Collaborator

@jingy-li jingy-li commented Sep 19, 2024

Transfer offsetRecord and storeVersionState during blob transfer

Main code changes:

  1. For P2PFileTransferClientHandler:
    • No longer use inputStream to determine if the transfer is complete. Instead, use BlobTransferPartitionMetadata, which is also needed for updating the offsetRecord later.
    • For the responses, distinguish between file and metadata types. If it's a file type, write the file; if it's a metadata type, use it to update the metadata.
    • In bootstrapFromBlobs, once the metadata is transferred, update the offsetRecord in storageMetadataService and update the SVS via StoreVersionStateSyncer.
  2. For P2PFileTransferServerHandler:
    • After P2PFileTransferServerHandler sends the file and before sending the BLOB_TRANSFER_COMPLETED flag, the server handler sends metadata record information with the partition's latest offset and store version state, retrieved from StorageMetadataService.

Minor code change:

  • Move the blob transfer client/server from venice-common to davinci, because it will use storageMetadataService to get the necessary offset/SVS information. Otherwise, it will cause a cyclic dependency.

How was this PR tested?

Unit test and integration test.
Screenshot 2024-09-19 at 10 24 46 AM

Does this PR introduce any user-facing changes?

  • No. You can skip the rest of this section.
  • Yes. Make sure to explain your proposed changes and call out the behavior change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant