Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lochen/pipeline output dataset #1645

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Lochen/pipeline output dataset #1645

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1c4959e
change pipeline data to pipeline output dataset
cloga Nov 24, 2021
96a8e06
make change to use pipeline_output_dataset
cloga Nov 25, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 15 additions & 6 deletions ...hine-learning-pipelines/intro-to-pipelines/aml-pipelines-with-data-dependency-steps.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,9 @@
"- **output_name:** Name of the output\n",
"- **output_mode:** Specifies \"upload\" or \"mount\" modes for producing output (default: mount)\n",
"- **output_path_on_compute:** For \"upload\" mode, the path to which the module writes this output during execution\n",
"- **output_overwrite:** Flag to overwrite pre-existing data"
"- **output_overwrite:** Flag to overwrite pre-existing data\n",
"\n",
"As PipelineData is using [DataReference](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.data_reference.datareference?view=azure-ml-py) to represent the data, which is not the recommanded approch, we will recommand you to use [pipeline_output_dataset](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipeline_output_dataset?view=azure-ml-py) instead, once promoted to an Azure Machine Learning dataset, it will also be consumed as a Dataset instead of a DataReference in subsequent steps."
]
},
{
Expand All @@ -252,8 +254,12 @@
"# is_directory=None)\n",
"\n",
"# Naming the intermediate data as processed_data1 and assigning it to the variable processed_data1.\n",
"processed_data1 = PipelineData(\"processed_data1\",datastore=def_blob_store)\n",
"print(\"PipelineData object created\")"
"# Promote pipelinedata to pipeline_output_dataset, which will use dataset instead of data reference \n",
"\n",
"from azureml.pipeline.core.pipeline_output_dataset import PipelineOutputFileDataset\n",
"\n",
"processed_data1 = PipelineOutputFileDataset(PipelineData(\"processed_data1\",datastore=def_blob_store))\n",
"print(\"PipelineOutputFileDataset object created\")"
]
},
{
Expand Down Expand Up @@ -544,10 +550,13 @@
"Azure ML"
],
"friendly_name": "Azure Machine Learning Pipelines with Data Dependency",
"interpreter": {
"hash": "3e9e0e270b75c5e6da2e22113ba4f77b864d68f95da6601809c29e46c73ae6bb"
},
"kernelspec": {
"display_name": "Python 3.6",
"language": "python",
"name": "python36"
"name": "python3"
},
"language_info": {
"codemirror_mode": {
Expand All @@ -559,7 +568,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.7"
"version": "3.7.8"
},
"order_index": 2,
"star_tag": [
Expand All @@ -572,4 +581,4 @@
},
"nbformat": 4,
"nbformat_minor": 2
}
}