You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We have implemented create and rename recovery using client transaction IDs over the DFS endpoint (HADOOP-19450 [ABFS] Rename/Create path idempotency client-level resolution - ASF JIRA). Since the backend changes were not fully rolled out, we initially implemented the changes with the flag disabled. With this update, we aim to enable the flag, which will start sending client transaction IDs. In case of a failure, we will attempt to recover from the failed state if possible. Here are the detailed steps and considerations for this process:
Implementation Overview: We introduced a mechanism for create and rename recovery via client transaction IDs to enhance reliability and data integrity over the DFS endpoint. This change was initially flagged as disabled due to incomplete backend rollouts.
Current Update: With the backend changes now in place, we are ready to enable the flag. This will activate the sending of client transaction IDs, allowing us to track and manage transactions more effectively.
Failure Recovery: The primary advantage of enabling this flag is the potential for recovery from failed states. If a transaction fails, we can use the client transaction ID to attempt a recovery, minimizing data loss and ensuring continuity.
Next Steps: We will proceed with enabling the flag and closely monitor the system's performance. Any issues or failures will be documented and addressed promptly to ensure a smooth transition.
The reason will be displayed to describe this comment to others. Learn more.
Unauthorized Blob Overwrite error we only get when the destination file already exists. For invalid permissions we have separate errors thrown. We can remove the "unauthorized overwrite" parts here.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Jira: https://issues.apache.org/jira/browse/HADOOP-19497
We have implemented create and rename recovery using client transaction IDs over the DFS endpoint (HADOOP-19450 [ABFS] Rename/Create path idempotency client-level resolution - ASF JIRA). Since the backend changes were not fully rolled out, we initially implemented the changes with the flag disabled. With this update, we aim to enable the flag, which will start sending client transaction IDs. In case of a failure, we will attempt to recover from the failed state if possible. Here are the detailed steps and considerations for this process:
Implementation Overview: We introduced a mechanism for create and rename recovery via client transaction IDs to enhance reliability and data integrity over the DFS endpoint. This change was initially flagged as disabled due to incomplete backend rollouts.
Current Update: With the backend changes now in place, we are ready to enable the flag. This will activate the sending of client transaction IDs, allowing us to track and manage transactions more effectively.
Failure Recovery: The primary advantage of enabling this flag is the potential for recovery from failed states. If a transaction fails, we can use the client transaction ID to attempt a recovery, minimizing data loss and ensuring continuity.
Next Steps: We will proceed with enabling the flag and closely monitor the system's performance. Any issues or failures will be documented and addressed promptly to ensure a smooth transition.