Skip to content
This repository was archived by the owner on Feb 27, 2025. It is now read-only.

Assessing the risk of duplicated entries for BEST_EFFORT reliabilityLevel #252

Open
jcblancomartinez opened this issue Jan 4, 2024 · 1 comment

Comments

@jcblancomartinez
Copy link

Hi,

From here:

Implements the BEST_EFFORT write strategy for Single Instance. All executors insert into a user specified table directly. Write to table is not transactional and may results in duplicates in executor restart scenarios.

I'm not sure I understand when we could end up with duplicates when using BEST_EFFORT reliabilityLevel.

For each Spark's dataframe partition we call savePartition and savePartition only commits the transaction after SQLServerBulkCopy.writeToServer has succeded.

Thanks.

@jcblancomartinez
Copy link
Author

jcblancomartinez commented Jan 4, 2024

@shivsood @luxu1-ms could you please help here?

Thanks.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant