Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Python] Add recipe for appending/replacing data set partitions #278

Open
assignUser opened this issue Dec 5, 2022 · 0 comments
Open

[Python] Add recipe for appending/replacing data set partitions #278

assignUser opened this issue Dec 5, 2022 · 0 comments

Comments

@assignUser
Copy link
Member

The pyarrow docs for the exisiting_data_behavior param contains this hint:

This behavior, in combination with a unique basename_template for each write, will allow for an append workflow.

‘delete_matching’ is useful when you are writing a partitioned dataset. The first time each partition directory is encountered the entire directory will be deleted. This allows you to overwrite old partitions completely.

A fully formed recipe for this would be nice as this hint is a bit hidden.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@assignUser