Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Time-travel & Incremental API to support more timestamp formats #261

Open
xushiyan opened this issue Jan 23, 2025 · 0 comments
Open

Time-travel & Incremental API to support more timestamp formats #261

xushiyan opened this issue Jan 23, 2025 · 0 comments
Labels
feature good first issue Good for newcomers
Milestone

Comments

@xushiyan
Copy link
Member

xushiyan commented Jan 23, 2025

Description of the improvement

Currently Table::read_incremental_records() expects start and end timestamps to be in the Hudi timeline format, i.e., yyyyMMddHHmmSSSSS or the legacy yyyyMMddHHmmSS. We should support parsing more timestamp formats.

Sames goes for Table::read_snapshot_as_of() and other *as_of() APIs.

Expected behavior

  • To support parsing strings in the form of epoch time (including seconds, milliseconds, microseconds, nanoseconds) and convert to Hudi timeline format for further processing.

  • To support parsing strings in the form of ISO8601 format like

    • 2019-01-23T12:34:56.123456789+00:00
    • 2019-01-23T12:34:56.123456+00:00
    • 2019-01-23T12:34:56.123+00:00
    • 2019-01-23T12:34:56+00:00
    • 2019-01-23T12:34:56Z and other precisions like above
    • 2019-01-23

Additional context

The conversion to Hudi timeline format should consider the hudi table's timeline timezone config.

@xushiyan xushiyan added good first issue Good for newcomers feature labels Jan 23, 2025
@xushiyan xushiyan added this to the release-0.4.0 milestone Jan 23, 2025
@xushiyan xushiyan changed the title Incremental API to support more timestamp formats Time-travel & Incremental API to support more timestamp formats Feb 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature good first issue Good for newcomers
Projects
Status: Todo
Development

No branches or pull requests

1 participant