feat: Switch geodata providers #7393

camdecoster · 2025-03-20T03:12:59Z

Description

Adds scripts to build topojson from UN sourced data and updates build process to run these scripts.

Closes #7334

Changes

Adds script for downloading shapefiles
Adds script for converting shapefiles into topojson
Updates topojson path references

Testing

Run npm run build_topojson and make sure the script completes successfully
Try loading the maps in ./dist/topojson in Mapshaper
Open the test dashboard
Try loading some of the geo plots and look for errors

Notes

The new maps are built from this UN data source for countries, coastlines, and lands layers. The oceans, lakes, rivers, and subunits layers are built from Natural Earth Data.
The current maps come from sane-topojson, which is entirely derived from Natural Earth Data
I haven't been able to verify the resolution of the data from the UN. There's a reference here to the "UNGeodata 25 million scale", but I haven't been able to confirm what that means. As such, the labels '50m' and '110m' probably aren't accurate. But they need to remain as they are to not break references.
Mapshaper can't handle antimeridian cutting, so features that cross the antimeridian can look weird (source)
I want to credit sane-topojson for some of the design of these scripts. I'm attempting to create a (mostly) drop in replacement, so it made sense to follow those conventions.

TODO

devtools/test_dashboard/devtools.js

tasks/topojson/process_geodata.mjs

ndrezn · 2025-03-20T13:43:09Z

tasks/topojson/get_geodata.mjs

+import config from './config.json' assert { type: 'json' };
+
+try {
+    // Download data from UN


As these origin URLs may change (link rot), I worry that this piece of the script could become out of date quickly. I'm not sure there's too much we can do here but I might recommend committing the current source file (but not adding it to the dist/. Maybe someone else has a better idea.

That seems reasonable. The script could be updated to look for the file first and only download if it doesn't exist.

@marthacryan do you have an opinion or ideas on this? What's the ideal dev UX here?

FYI, in 3b47c67 I added the archive from the UN.

Co-authored-by: Nathan Drezner <[email protected]>

gvwilson · 2025-03-25T14:05:52Z

Sync with this branch.
npm i to install/update packages.
npm run build_topojson to build JSON files.
Go to mapshaper.org.
Load europe_50m.json.
Initially looks like the lakes are misplaced (for example, the Caspian Sea seems to overlap the Russian border in a weird way in the screenshot).
On closer inspection, maybe this is valid: I'm unable to find a map showing how national borders cross the Caspian Sea, and Russia might very well have a claim to the northwest section.
The three lakes in the north check out: one is in Sweden, the other two are near St. Petersburg.

gvwilson · 2025-03-25T14:08:10Z

Q for @camdecoster : did the old usa map have the states? as you say, it's not much use without that…

gvwilson · 2025-03-25T14:50:48Z

Trying to get npm run test-jasmine to work with the new topojson files, it looks like I'm tripping over a configuration issue.

Original error message complained about not being able to find ../../../dist/topojsonworld_110m.json (note the missing '/' between 'topojson' and 'world').
I modified getTopojsonPath in src/lib/topojson_utils.js as follows to insert the '/' if required:

 topojsonUtils.getTopojsonPath = function(topojsonURL, topojsonName) {
-    return topojsonURL + topojsonName + '.json';
+    if (topojsonURL.endsWith('/')) {
+       return topojsonURL + topojsonName + '.json';
+    } else {
+       return topojsonURL + '/' + topojsonName + '.json';
+    }
 };

I also modified test/jasmine/karma.conf.js as follows because we're getting the JSON files from the local dist directory not the installed sane-topojson package:

-var pathToSaneTopojsonDist = path.join(__dirname, '..', '..', 'node_modules', 'sane-topojson', 'dist');
+var pathToTopojsonDist = path.join(__dirname, '..', '..', 'dist', 'topojson');

// ... and further down, edit to match the new variable name

-        {pattern: pathToSaneTopojsonDist + '/**', included: false, watched: false, served: true}
+        {pattern: pathToTopojsonDist + '/**', included: false, watched: false, served: true}

This still produces error messages like the one shown below saying that the JSON files can't be found:

	Failed: plotly.js could not find topojson file at ../../../dist/topojson/world_110m.json. Make sure the *topojsonURL* plot config option is set properly.
	Error: plotly.js could not find topojson file at ../../../dist/topojson/world_110m.json. Make sure the *topojsonURL* plot config option is set properly.
	    at Object.<anonymous> (/Users/gvwilson/plotly/plotly.js/src/plots/geo/geo.js:137:35 <- /private/var/folders/w2/l51fjbjd25n9zbwkz9fw9jp00000gn/T/0c95a0b81672d3a2e3f59fc0534657ef-bundle.js:147433:31)
	    at Object.event (/Users/gvwilson/plotly/plotly.js/node_modules/@plotly/d3/d3.js:504:42 <- /private/var/folders/w2/l51fjbjd25n9zbwkz9fw9jp00000gn/T/0c95a0b81672d3a2e3f59fc0534657ef-bundle.js:4875:48)
	    at XMLHttpRequest.respond (/Users/gvwilson/plotly/plotly.js/node_modules/@plotly/d3/d3.js:1951:24 <- /private/var/folders/w2/l51fjbjd25n9zbwkz9fw9jp00000gn/T/0c95a0b81672d3a2e3f59fc0534657ef-bundle.js:6326:30)
25 03 2025 10:40:03.170:WARN [web-server]: 404: /dist/topojson/world_110m.json
[note: the message above is then repeated several times]

I've tried variations on the path in test/jasmine/tests/geo_test.js specified by this line:

Plotly.setPlotConfig({ topojsonURL: '../../../dist/topojson' });

Adding an extra .. or removing one of the ones that's there doesn't affect the outcome.

camdecoster · 2025-03-25T14:59:26Z

@gvwilson yes the old maps had 'subunits' layers for a few regions (USA and Brazil did for sure). The UN maps don't include that information. Thanks for looking into the tests. I'm working on an update to get the tests working but I haven't pushed that yet. I'll try to get that out later today.

ndrezn · 2025-03-25T15:05:11Z

@etpinard when you have a chance could you give this PR a look? Would love your eyes. Thank you!

In the short term what was the source for US states/Canadian provinces in the old dataset?

Refactor to simplify scripts Switch to UN/NE geodata hybrid Save UN geodata to archive Remove extra info from topojson Add centroids to geojson Use 'simplify' to create 110m maps

…data-providers

src/assets/geo_assets.js

emilykl · 2025-04-22T16:00:41Z

@camdecoster One small suggestion: Either add tasks/topojson/un_geodata_simplified.zip to .gitignore, OR delete the file as part of cleanup from the get_geodata task.

camdecoster · 2025-04-22T17:21:08Z

@emilykl thanks for pointing that out. I actually meant to ask if that should be saved in the repo. Maybe that's not necessary, but if the files change in the future then the build process could break. Granted, we're not saving the Natural Earth shapefiles, so that is another potential point for things to break. Ultimately the question is, should we save the input files in the repo or download them each time when building the library?

emilykl · 2025-04-22T17:35:38Z

@camdecoster It's a good question... curious what @gvwilson @ndrezn think.

Presumably both the UN data and the Natural Earth data will be updated periodically as the world changes (is my assumption correct? or are we referencing a static URL that will never change?) and we will want to pull in the updated data.

On the other hand that's not necessarily a step that we want or need to be doing on EVERY build necessarily.

Are there any steps in the build process that you think are particularly sensitive to changes in the data? How much work would it be to fix the build process if/when the data changes?

gvwilson · 2025-04-22T18:42:59Z

Decision for now is to download the data afresh each time the build runs; we'll come back in a week or two and modify the build to cache the data in the repo. cc @emilykl @camdecoster

camdecoster added 4 commits March 19, 2025 18:16

Add new dev dependencies

e8d763d

Add build scripts

7f7b690

Update some topojson references

6b596c0

Add npm build scripts

353bf94

gvwilson assigned ndrezn Mar 20, 2025

gvwilson added feature something new P1 needed for current cycle labels Mar 20, 2025

camdecoster added 2 commits March 20, 2025 07:13

Add 'usa' config

fc6022f

Add clipping bounds for Antarctica

2910eea

camdecoster changed the title ~~Switch geodata providers~~ feat: Switch geodata providers Mar 20, 2025

ndrezn reviewed Mar 20, 2025

View reviewed changes

ndrezn requested a review from emilykl March 20, 2025 17:03

camdecoster and others added 6 commits March 20, 2025 11:40

Apply suggestions from code review

5d66dad

Co-authored-by: Nathan Drezner <[email protected]>

Add geodata archive, update get_geodata script paths

3b47c67

Switch config file to JS

b434656

Switch coastlines layer source

a4d0b21

Add Oceania clipping bounds

e257b7b

Switch layer name to 'ocean'

1154ff1

camdecoster added 7 commits March 25, 2025 20:42

Update test config

86d5b00

Switch to UN/NE geodata hybrid

86f048c

Refactor to simplify scripts Switch to UN/NE geodata hybrid Save UN geodata to archive Remove extra info from topojson Add centroids to geojson Use 'simplify' to create 110m maps

Merge remote-tracking branch 'origin/master' into cam/7334/switch-geo…

ac61095

…data-providers

Fix zip/unzip of UN geojson

f4c3d48

Fix saving 110m maps

22a8e0a

Update dist with new maps

247f925

Update geo tests

3cf9730

camdecoster added 2 commits April 7, 2025 21:28

Update cibuild npm script

24e8558

Add draftlog

b2ae532

camdecoster marked this pull request as ready for review April 8, 2025 18:28

emilykl reviewed Apr 8, 2025

View reviewed changes

src/assets/geo_assets.js Show resolved Hide resolved

camdecoster added 3 commits April 14, 2025 18:38

Remove sane-topojson package

85ad69c

Add small fixes

b903b9f

Remove log statements

e74c65c

gvwilson assigned camdecoster Apr 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Switch geodata providers #7393

feat: Switch geodata providers #7393

camdecoster commented Mar 20, 2025 •

edited

Loading

ndrezn Mar 20, 2025

camdecoster Mar 20, 2025

ndrezn Mar 21, 2025

camdecoster Mar 22, 2025

gvwilson commented Mar 25, 2025

gvwilson commented Mar 25, 2025

gvwilson commented Mar 25, 2025

camdecoster commented Mar 25, 2025

ndrezn commented Mar 25, 2025

emilykl commented Apr 22, 2025

camdecoster commented Apr 22, 2025

emilykl commented Apr 22, 2025

gvwilson commented Apr 22, 2025

feat: Switch geodata providers #7393

Are you sure you want to change the base?

feat: Switch geodata providers #7393

Conversation

camdecoster commented Mar 20, 2025 • edited Loading

Description

Changes

Testing

Notes

TODO

ndrezn Mar 20, 2025

Choose a reason for hiding this comment

camdecoster Mar 20, 2025

Choose a reason for hiding this comment

ndrezn Mar 21, 2025

Choose a reason for hiding this comment

camdecoster Mar 22, 2025

Choose a reason for hiding this comment

gvwilson commented Mar 25, 2025

gvwilson commented Mar 25, 2025

gvwilson commented Mar 25, 2025

camdecoster commented Mar 25, 2025

ndrezn commented Mar 25, 2025

emilykl commented Apr 22, 2025

camdecoster commented Apr 22, 2025

emilykl commented Apr 22, 2025

gvwilson commented Apr 22, 2025

camdecoster commented Mar 20, 2025 •

edited

Loading