Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Metric] Feature title关于datavines“及时性检查”(英文freshness)在使用过程中发现的问题 #489

Open
1 of 2 tasks
18859108815 opened this issue Jan 2, 2025 · 0 comments

Comments

@18859108815
Copy link

Search before asking

  • I had searched in the feature and found no similar feature requirement.

Description

datavines已支持“及时性检查”(英文freshness),在使用过程中发现如下几个问题:

  1. 目前要求必须选“列”,且该列的字段类型必须为“datetime”类型,而生产环境中大量存在着“数据时间”类型为varchar或者int类型,此时无法使用“及时性检查”规则。
  2. 在生成的脚本文件中,会用上date_format函数,要求对应的数据源需支持date_format函数,例如mysql支持该函数,而oracle不支持,就会导致oracle数据库不能用“及时性检查”规则。
    "sql": "select count(1) as actual_value_0544943c from ( select * from \"datasa\".\"A_5G2B_ASSESS_WR_D\" where (DATE_FORMAT(\"SDATE\", 'YYYYMMDD') <= DATE_FORMAT('20250102', 'YYYYMMDD') ) AND (DATE_FORMAT(\"SDATE\", 'YYYYMMDD') >= DATE_FORMAT('20250101', 'YYYYMMDD')) ) t"
  3. “及时性检查”规则中要求要填写“开始时间”和“结束时间”,如果填写具体的时间的话,那就无法实现实现数据时间随着自然时间的增加而自动调整,如果使用系统预置的时间,系统只支持时间偏移量为“天”,如果我要获取n小时或者n分钟前的时间,则无法获取,这个需求的使用场景是小时粒度的数据,数据时延预计是120分钟,当前的内置时间无法支持该场景。
  4. 其他的检查和及时性检查存在依赖关系,比如完整性检查的前期是需要通过及时性检查,这个需求是否可以结合DolphinScheduler来实现,通过“工作流”来规范各数据质量作业的执行顺序。

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!
@18859108815 18859108815 changed the title [Feature][Module Name] Feature title [Feature][Module Name] Feature title关于datavines“及时性检查”(英文freshness)在使用过程中发现的问题 Jan 2, 2025
@zixi0825 zixi0825 changed the title [Feature][Module Name] Feature title关于datavines“及时性检查”(英文freshness)在使用过程中发现的问题 [Feature][Metric] Feature title关于datavines“及时性检查”(英文freshness)在使用过程中发现的问题 Jan 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant