Update for latest ADX version #2
Conversation
|
Fixes dsdk version and downloading sample data. |
| "- Expand the \"Data set overview\" box near the top to reveal the Data set ID.\n", | ||
| "\n", | ||
| "_**Replace the `dataset_id` parameter below with your particular ID then run the notebook.**_\n", | ||
| "The use license will be in your `PURESKILLGG_TOME_DS_COLLECTION_PATH` named `license.pdf`. You must agree to these terms to use the data.\n", |
There was a problem hiding this comment.
| "The use license will be in your `PURESKILLGG_TOME_DS_COLLECTION_PATH` named `license.pdf`. You must agree to these terms to use the data.\n", | |
| "The use license will be in your `PURESKILLGG_TOME_DS_COLLECTION_PATH` named `LICENSE.pdf`. You must agree to these terms to use the data.\n", |
Same for README.md in the zip file.
| @@ -89,23 +69,36 @@ | |||
| "outputs": [], | |||
| "source": [ | |||
| "import os\n", | |||
| "from pureskillgg_dsdk.exchange import download_dataexchange_dataset_revision" | |||
| "import requests\n", | |||
There was a problem hiding this comment.
Interesting, when I looked up how to do this I was suggested to use import urllib and urllib.request. Is requests also in std lib? Two ways to do the same thing in Python? 😲
There was a problem hiding this comment.
my google search led me to this and i didn't look more into it haha https://www.codingem.com/python-download-file-from-url/
Requests is mentioned in the top yellow box here https://docs.python.org/3/library/urllib.request.html#module-urllib.request
There was a problem hiding this comment.
Yea, apparently I used the legacy interface: https://docs.python.org/3/library/urllib.request.html#urllib.request.urlretrieve at https://github.com/pureskillgg/dsdk/blob/6f7dcc064e6a4778fc268e2eb0c191fbd5f6d965/pureskillgg_dsdk/exchange/dataset.py#L75
Anyway, we will need to add this with poetry yea?
| "download_dataexchange_dataset_revision(output_path, dataset_id)" | ||
| " response = requests.get(url)\n", | ||
| " with open(output_filename, \"wb\") as f:\n", | ||
| " f.write(response.content)\n", |
There was a problem hiding this comment.
We can just skip the part where we save the zip file to disk: https://stackoverflow.com/a/2463819
There was a problem hiding this comment.
Nice. Though now I can't check if the zip exists to see if we've already downloaded it 🤔
There was a problem hiding this comment.
This is such a small amount of data I'm not sure it matters
There was a problem hiding this comment.
Also note that the way you've written this, if there was an error extracting the zip file and they reran the loop it would still assume the zip file was extracted correctly. So I recommend just not making this one "smart"
There was a problem hiding this comment.
The old one took 20 minutes to download, this one takes less than 10 seconds so it is trivial to overwrite, i guess :)
No description provided.