Update cpueff_stepchain_goweb.py

nikodemas · web-flow · commit 1cbd49a4c7e4 · 2024-03-08T09:31:15.000+01:00
Change date delta in cpueff
diff --git a/cpueff-goweb/spark/cpueff_stepchain_goweb.py b/cpueff-goweb/spark/cpueff_stepchain_goweb.py
@@ -236,7 +236,7 @@ def main(start_date, end_date, hdfs_out_dir, last_n_days):
         spark.read.option("basePath", _DEFAULT_HDFS_FOLDER)
         .json(
             get_candidate_files(
-                start_date, end_date, spark, base=_DEFAULT_HDFS_FOLDER, day_delta=2
+                start_date, end_date, spark, base=_DEFAULT_HDFS_FOLDER, day_delta=1
             )
         )
         .select(["data.*", "metadata.timestamp"])

Original file line number	Diff line number	Diff line change
`@@ -236,7 +236,7 @@ def main(start_date, end_date, hdfs_out_dir, last_n_days):`
`236`	`236`	`spark.read.option("basePath", _DEFAULT_HDFS_FOLDER)`
`237`	`237`	`.json(`
`238`	`238`	`get_candidate_files(`
`239`		`- start_date, end_date, spark, base=_DEFAULT_HDFS_FOLDER, day_delta=2`
	`239`	`+ start_date, end_date, spark, base=_DEFAULT_HDFS_FOLDER, day_delta=1`
`240`	`240`	`)`
`241`	`241`	`)`
`242`	`242`	`.select(["data.*", "metadata.timestamp"])`