Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update for latest dsdk #7

Merged
merged 16 commits into from
Jun 12, 2022
Merged

Update for latest dsdk #7

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
d943997
Update setup link, header tome name, root path ends
billfreeman44 Jun 12, 2022
f6c08c7
Repair default values for notebook 2
billfreeman44 Jun 12, 2022
a4eeddf
Remove path_depth and fix subheader tome name
billfreeman44 Jun 12, 2022
114d87e
Add note about default tome names
billfreeman44 Jun 12, 2022
6b9215d
update csgo dsdk and dsdk
billfreeman44 Jun 12, 2022
ca32c3a
Mention inclusive/exclusive for dates
billfreeman44 Jun 12, 2022
4a6cc59
Use proper tome name. specifiy header tome b/c bug
billfreeman44 Jun 12, 2022
135c68d
Use correct tome name convention
billfreeman44 Jun 12, 2022
8d75c41
Use the same example path as notebook 1
billfreeman44 Jun 12, 2022
d52838e
change path to csds to path to root
billfreeman44 Jun 12, 2022
224297c
User sets header and tome paths. Warning for bad name.
billfreeman44 Jun 12, 2022
2adf784
Update dsdk and csgodsdk
billfreeman44 Jun 12, 2022
13b9522
Double check if dir exists, if not make it.
billfreeman44 Jun 12, 2022
25092f6
Update 2 - Download data.ipynb
billfreeman44 Jun 12, 2022
f7bcd10
Remove hotfix for bug from dsdk
billfreeman44 Jun 12, 2022
111ef90
Fix convention typo
billfreeman44 Jun 12, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions notebooks/tutorial/1 - Setup.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
"source": [
"## Setup your AWS credentials\n",
"\n",
"Follow the [guide on setting up AWS credentials](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html) to properly download things from the AWS Data Exchange."
"Follow the [guide on setting up AWS credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html#configuration) to properly download things from the AWS Data Exchange."
]
},
{
Expand Down Expand Up @@ -63,13 +63,13 @@
"import os\n",
"import platform\n",
"\n",
"header_name = 'header_tome'\n",
"header_name = 'header_tome.2022-05-15,2022-05-15'\n",
"ds_type = 'csds'\n",
"\n",
"if collection_path == 'PATH/TO/TOMES':\n",
" collection_path = os.path.join('..','..','tmp','tomes')\n",
" collection_path = os.path.join('..','..','tmp','tome_root')\n",
"if ds_collection_path == 'PATH/TO/CSDS':\n",
" ds_collection_path = os.path.join('..','..','tmp','data')\n",
" ds_collection_path = os.path.join('..','..','tmp','csds_root')\n",
" \n",
"collection_path = os.path.abspath(collection_path)\n",
"ds_collection_path = os.path.abspath(ds_collection_path)\n",
Expand Down
10 changes: 7 additions & 3 deletions notebooks/tutorial/3 - Make header tome.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@
"outputs": [],
"source": [
"if not header_loader.exists:\n",
" header_loader = curator.create_header_tome(path_depth=7)"
" header_loader = curator.create_header_tome()"
]
},
{
Expand All @@ -110,7 +110,11 @@
"source": [
"## Make subheaders too\n",
"\n",
"You might want to analyze players on a specific map, rank, or platform. You can create \"subheaders\" that are a filtered view of the main header. The `create_subheader_tome` will create the subheader with the specified filter applied to the header tome."
"You might want to analyze players on a specific map, rank, or platform. You can create \"subheaders\" that are a filtered view of the main header. The `create_subheader_tome` will create the subheader with the specified filter applied to the header tome.\n",
"\n",
"By convention, the tome names are in this format: `tome_name.start-date,end-data.comment`\n",
"\n",
"The `start-date` (inclusive) and `end-date` (exclusive) are the beginning and end of the data included in that tome."
]
},
{
Expand All @@ -123,7 +127,7 @@
"def map_name_selector(map_name):\n",
" return lambda df: df['map_name']==map_name\n",
"\n",
"subheader_loader = curator.create_subheader_tome('subheader_dust2', map_name_selector('de_dust2'))"
"subheader_loader = curator.create_subheader_tome('subheader_dust2.2022-05-15,2022-05-15', map_name_selector('de_dust2'))"
]
},
{
Expand Down
6 changes: 4 additions & 2 deletions notebooks/tutorial/5 - Create tome.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,8 +96,10 @@
"outputs": [],
"source": [
"# Initialize our \"footsteps_by_rank\" tome\n",
"footsteps_tome_name = 'footsteps_by_rank.2022-05-15,2022-05-15'\n",
"tomer = curator.make_tome(\n",
" 'footsteps_by_rank',\n",
" footsteps_tome_name,\n",
" header_tome_name=os.environ.get(\"PURESKILLGG_TOME_DEFAULT_HEADER_NAME\"),\n",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did we need to add this line? Doesn't it use that env var automatically?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug introduced when i made the copier its own class. Fix is here, just gotta upgrade dsdk version. pureskillgg/dsdk@9907100

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in latest DSDK

" ds_reading_instructions=[\n",
" {\n",
" \"channel\": 'player_footstep',\n",
Expand Down Expand Up @@ -137,7 +139,7 @@
"metadata": {},
"outputs": [],
"source": [
"df = curator.get_dataframe('footsteps_by_rank')"
"df = curator.get_dataframe(footsteps_tome_name)"
]
},
{
Expand Down
5 changes: 3 additions & 2 deletions notebooks/tutorial/6 - Train data science models.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,9 @@
"metadata": {},
"outputs": [],
"source": [
"df = curator.get_dataframe('footsteps_by_rank')\n",
"keyset = curator.get_keyset('footsteps_by_rank')"
"footsteps_tome_name = 'footsteps_by_rank.2022-05-15,2022-05-15'\n",
"df = curator.get_dataframe(footsteps_tome_name)\n",
"keyset = curator.get_keyset(footsteps_tome_name)"
]
},
{
Expand Down
4 changes: 2 additions & 2 deletions notebooks/tutorial/7 - Getting data from the ADX.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -619,10 +619,10 @@
"There are many many ways to do this so we won't list them all here. We generally sync one month at a time with the AWS CLI like this:\n",
"\n",
"```\n",
"aws s3 sync s3://my-bucket/csds/2022/04/ /path/to/working/dir/csds/2022/04\n",
"aws s3 sync s3://my-bucket/csds/2022/04/ /PATH/TO/CSDS/csds/2022/04\n",
"```\n",
"\n",
"It is **CRITICALLY IMPORTANT** to maintain the path structure. In the example above, you could change the `/path/to/working/dir` but the `csds/2022/04` is not something that you may change. The reader will not work because the folder structure is part of the key used to read each file."
"It is **CRITICALLY IMPORTANT** to maintain the path structure. In the example above, you could change the `/PATH/TO/CSDS` but the `csds/2022/04` is not something that you may change. The reader will not work because the folder structure is part of the key used to read each file."
]
},
{
Expand Down
Loading