Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate to external ETL regions dataset #4534

Merged
merged 2 commits into from
Feb 6, 2025
+9 −4
Merged

Migrate to external ETL regions dataset #4534

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b40b6f8
Update update.ts
lucasrodes Feb 6, 2025
9da9b44
enhance: exclude income groups for now
marcelgerber Feb 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions devTools/regionsUpdater/update.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import _ from "lodash"

const ETL_REGIONS_URL =
process.env.ETL_REGIONS_URL ||
"https://catalog.ourworldindata.org/grapher/regions/latest/regions/regions.csv",
"https://catalog.ourworldindata.org/external/owid_grapher/latest/regions/regions.csv",
GEO_JSON_URL =
"https://raw.githubusercontent.com/alexabruck/worldmap-sensitive/master/dist/world.geo.json",
GRAPHER_ROOT = __dirname.replace(/\/(itsJustJavascript\/)?devTools.*/, ""),
Expand All @@ -31,7 +31,10 @@ const ETL_REGIONS_URL =
MKD: ["Macedonia"],
SWZ: ["Swaziland"],
USA: ["US", "USA"],
}
},
// we want to exclude income groups for now, until we can properly display the user's
// income group in the UI
REGIONS_TO_EXCLUDE = ["OWID_HIC", "OWID_UMC", "OWID_LMC", "OWID_LIC"]

interface Entity {
code: string
Expand Down Expand Up @@ -203,8 +206,10 @@ async function main() {
transform: csvToJson,
})

// strip out empty rows and make sure entities are sorted
data = _.sortBy(data, "code").filter((c: any) => !!c.code)
// strip out empty and excluded rows and make sure entities are sorted
data = _.sortBy(data, "code")
.filter((c: any) => !!c.code)
.filter((c: any) => !REGIONS_TO_EXCLUDE.includes(c.code))

const entities = _.map(data as Entity[], (entity) => {
// drop redundant attrs
Expand Down