HOL02-AML/HOL-Azure Machine Learning service.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Copyright (c) Microsoft Corporation. All rights reserved."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![alt text](https://www.microsoftevents.com/accounts/register123/microsoft/msft-v1/c-and-e-v2/events/ce2-ce-2c-mec0028133/Azure%20Academy%20banner_Data.png \"Logo Title Text 1\")\n",
        "\n",
        "# HOL02: Azure Machine Learning serice"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This lab guides you through Azure Machine Learning service - creation & setup, building experiment and train model (also use automated machine learning technique).\n",
        "\n",
        "In this use case you build a regression model to predict NYC taxi fare prices. \n",
        "This process accepts training data and configuration settings, and automatically iterates through combinations of different feature normalization/standardization methods, models, and hyperparameter settings to arrive at the best model.\n",
        "\n",
        "In this lab you learn the following tasks:\n",
        "\n",
        "* Create Azure Machine Learning Workspace\n",
        "* Download, transform, and clean data using Azure Open Datasets\n",
        "* Train an automated machine learning regression model\n",
        "* Calculate model accuracy\n",
        "\n",
        "If you don’t have an Azure subscription, create a free account before you begin. Try the [free or paid version](https://aka.ms/AMLFree) of Azure Machine Learning service today."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Prerequisites"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "* Existing AML Workspace - step can be found on GitHub repo for this lab [AzureAcademy-DataAnalyst-II-ML-AI-HOL01-AML.md](https://github.com/michalmar/azure-labs/blob/master/AzureAcademy-DataAnalyst-II-ML-AI-HOL01-AML.md)\n",
        "\n",
        "* check and update VM environemt\n",
        "\n",
        "`pip install --upgrade azureml-sdk[explain,automl,notebooks] azureml-opendatasets azureml-widgets \"urllib3==1.24\"`\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# PART 1: Download and prepare data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We will use [Azure Open Datasets](https://docs.microsoft.com/en-us/azure/open-datasets/overview-what-are-open-datasets) - curated public datasets that you can use to add scenario-specific features to machine learning solutions for more accurate models. Open Datasets are in the cloud on Microsoft Azure and are readily available to Azure Databricks, Machine Learning service, and Machine Learning Studio. You can also access the datasets through APIs and use them in other products, such as Power BI and Azure Data Factory.\n",
        "\n",
        "\n",
        "We will use particular dataset: [NYC Taxi & Limousine Commission - green taxi trip records](https://azure.microsoft.com/en-us/services/open-datasets/catalog/nyc-taxi-limousine-commission-green-taxi-trip-records/)\n",
        "\n",
        "\n",
        "\n",
        "Import the necessary packages. The Open Datasets package contains a class representing each data source (`NycTlcGreen` for example) to easily filter date parameters before downloading."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646662965751
        }
      },
      "outputs": [],
      "source": [
        "from azureml.opendatasets import NycTlcGreen\n",
        "import pandas as pd\n",
        "from datetime import datetime\n",
        "from dateutil.relativedelta import relativedelta\n",
        "\n",
        "%config IPCompleter.greedy=True"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Begin by creating a dataframe to hold the taxi data. When working in a non-Spark environment, Open Datasets only allows downloading one month of data at a time with certain classes to avoid `MemoryError` with large datasets. To download taxi data, iteratively fetch one month at a time, and before appending it to `green_taxi_df` randomly sample 2,000 records from each month to avoid bloating the dataframe. Then preview the data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# green_taxi_df = pd.read_csv(\"./data/taxi_raw_df.csv\")\n",
        "# green_taxi_df['lpepPickupDatetime'] = green_taxi_df['lpepPickupDatetime'].astype('datetime64[ns]')\n",
        "# # green_taxi_df.head(10)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646663152934
        }
      },
      "outputs": [],
      "source": [
        "green_taxi_df = pd.DataFrame([])\n",
        "start = datetime.strptime(\"1/1/2015\",\"%m/%d/%Y\")\n",
        "end = datetime.strptime(\"1/31/2015\",\"%m/%d/%Y\")\n",
        "\n",
        "for sample_month in range(12):\n",
        "    temp_df_green = NycTlcGreen(start + relativedelta(months=sample_month), end + relativedelta(months=sample_month)) \\\n",
        "        .to_pandas_dataframe()\n",
        "    green_taxi_df = green_taxi_df.append(temp_df_green.sample(2000))\n",
        "\n",
        "green_taxi_df.head(10)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646663954339
        }
      },
      "outputs": [],
      "source": [
        "green_taxi_df.to_parquet(\"./data/taxi_raw_df.parquet\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646663956280
        }
      },
      "outputs": [],
      "source": [
        "green_taxi_df = pd.read_parquet(\"./data/taxi_raw_df.parquet\")\n",
        "green_taxi_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now that the initial data is loaded, define a function to create various time-based features from the pickup datetime field. This will create new fields for the month number, day of month, day of week, and hour of day, and will allow the model to factor in time-based seasonality. \n",
        "\n",
        "Use the `apply()` function on the dataframe to iteratively apply the `build_time_features()` function to each row in the taxi data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646663967230
        }
      },
      "outputs": [],
      "source": [
        "def build_time_features(vector):\n",
        "    pickup_datetime = vector[0]\n",
        "    month_num = pickup_datetime.month\n",
        "    day_of_month = pickup_datetime.day\n",
        "    day_of_week = pickup_datetime.weekday()\n",
        "    hour_of_day = pickup_datetime.hour\n",
        "    \n",
        "    return pd.Series((month_num, day_of_month, day_of_week, hour_of_day))\n",
        "\n",
        "green_taxi_df[[\"month_num\", \"day_of_month\",\"day_of_week\", \"hour_of_day\"]] = green_taxi_df[[\"lpepPickupDatetime\"]].apply(build_time_features, axis=1)\n",
        "green_taxi_df.head(10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Remove some of the columns that you won't need for training or additional feature building."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646663972435
        }
      },
      "outputs": [],
      "source": [
        "columns_to_remove = [\"lpepPickupDatetime\", \"lpepDropoffDatetime\", \"puLocationId\", \"doLocationId\", \"extra\", \"mtaTax\",\n",
        "                     \"improvementSurcharge\", \"tollsAmount\", \"ehailFee\", \"tripType\", \"rateCodeID\", \n",
        "                     \"storeAndFwdFlag\", \"paymentType\", \"fareAmount\", \"tipAmount\"\n",
        "                    ]\n",
        "for col in columns_to_remove:\n",
        "    green_taxi_df.pop(col)\n",
        "    \n",
        "green_taxi_df.head(5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Cleanse data "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Run the `describe()` function on the new dataframe to see summary statistics for each field."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646663978982
        }
      },
      "outputs": [],
      "source": [
        "green_taxi_df.describe()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646663988442
        }
      },
      "outputs": [],
      "source": [
        "# Will allow us to embed images in the notebook\n",
        "%matplotlib inline\n",
        "import matplotlib.pyplot as plt\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646663992288
        },
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "plt.rcParams['figure.figsize'] = [16, 10]\n",
        "boxplot = green_taxi_df.boxplot(column=list(green_taxi_df.columns[1:-4]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "From the summary statistics, you see that there are several fields that have outliers or values that will reduce model accuracy. First filter the lat/long fields to be within the bounds of the Manhattan area. This will filter out longer taxi trips or trips that are outliers in respect to their relationship with other features. \n",
        "\n",
        "Additionally filter the `tripDistance` field to be greater than zero but less than 31 miles (the haversine distance between the two lat/long pairs). This eliminates long outlier trips that have inconsistent trip cost.\n",
        "\n",
        "Lastly, the `totalAmount` field has negative values for the taxi fares, which don't make sense in the context of our model, and the `passengerCount` field has bad data with the minimum values being zero.\n",
        "\n",
        "Filter out these anomalies using query functions, and then remove the last few columns unnecessary for training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664000276
        }
      },
      "outputs": [],
      "source": [
        "final_df = green_taxi_df.query(\"pickupLatitude>=40.53 and pickupLatitude<=40.88\")\n",
        "final_df = final_df.query(\"pickupLongitude>=-74.09 and pickupLongitude<=-73.72\")\n",
        "final_df = final_df.query(\"tripDistance>=0.25 and tripDistance<31\")\n",
        "final_df = final_df.query(\"passengerCount>0 and totalAmount>0\")\n",
        "\n",
        "columns_to_remove_for_training = [\"pickupLongitude\", \"pickupLatitude\", \"dropoffLongitude\", \"dropoffLatitude\"]\n",
        "for col in columns_to_remove_for_training:\n",
        "    final_df.pop(col)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Call `describe()` again on the data to ensure cleansing worked as expected. You now have a prepared and cleansed set of taxi, holiday, and weather data to use for machine learning model training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664004005
        }
      },
      "outputs": [],
      "source": [
        "final_df.describe()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664010709
        }
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "train, test = train_test_split(final_df, test_size=0.2, random_state=223)\n",
        "\n",
        "print(f'train:{len(list(train[\"vendorID\"]))} \\ntest: {len(list(test[\"vendorID\"]))}')\n",
        "\n",
        "final_df.to_csv(\"./data/taxi_final_df.csv\", index=False)\n",
        "train.to_csv(\"./data/taxi_final_df_train.csv\", index=False)\n",
        "test.to_csv(\"./data/taxi_final_df_test.csv\", index=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## PART 2: Train within notebook regression model with AML Serice"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In PART2 you train simple regression model within the notebook environement while logging metrics and output trhough AML service Experiment. Also you try to run single parameter sweep of the regression model.\n",
        "\n",
        "* first you create configure a connection to workspace\n",
        "* then run the simple training\n",
        "* lastly you run simple parameter sweep of a regression model\n",
        "* review results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664020076
        }
      },
      "outputs": [],
      "source": [
        "final_df = pd.read_csv(\"./data/taxi_final_df.csv\")\n",
        "# final_df.head(10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Configure workspace\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Create a workspace object from the existing workspace. A [Workspace](https://docs.microsoft.com/python/api/azureml-core/azureml.core.workspace.workspace?view=azure-ml-py) is a class that accepts your Azure subscription and resource information. It also creates a cloud resource to monitor and track your model runs. `Workspace.from_config()` reads the file **config.json** and loads the authentication details into an object named `ws`. `ws` is used throughout the rest of the code in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664138937
        }
      },
      "outputs": [],
      "source": [
        "import azureml\n",
        "from azureml.core.workspace import Workspace\n",
        "ws = Workspace.from_config()\n",
        "# print(f\"Name: {ws.name}, Resource group: {ws.resource_group}, Location: {ws.location}, Subscription: {ws.subscription_id}\")\n",
        "output = {}\n",
        "output['SDK version'] = azureml.core.VERSION\n",
        "output['Subscription ID'] = ws.subscription_id\n",
        "output['Workspace'] = ws.name\n",
        "output['Resource Group'] = ws.resource_group\n",
        "output['Location'] = ws.location\n",
        "# output['Run History Name'] = experiment_name\n",
        "pd.set_option('display.max_colwidth', -1)\n",
        "outputDf = pd.DataFrame(data = output, index = [''])\n",
        "outputDf.T"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Train locally within notebook"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Split the data into training and test sets by using the `train_test_split` function in the `scikit-learn` library. This function segregates the data into the x (**features**) data set for model training and the y (**values to predict**) data set for testing. The `test_size` parameter determines the percentage of data to allocate to testing. The `random_state` parameter sets a seed to the random generator, so that your train-test splits are deterministic."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664155156
        }
      },
      "outputs": [],
      "source": [
        "from sklearn.linear_model import Ridge\n",
        "from sklearn.metrics import mean_squared_error\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.externals import joblib\n",
        "\n",
        "import lightgbm as lgb\n",
        "\n",
        "\n",
        "y_df = final_df.pop(\"totalAmount\")\n",
        "x_df = final_df\n",
        "\n",
        "x_train, x_test, y_train, y_test = train_test_split(x_df, y_df, test_size=0.2, random_state=223)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's use scikit-learn to train a simple LightGBM regression model.  We use AML to record interesting information about the model in an Experiment.  An Experiment contains a series of trials called Runs.  During this trial we use AML in the following way:\n",
        "* We access an experiment from our AML workspace by name, which will be created if it doesn't exist\n",
        "* We use `start_logging` to create a new run in this experiment\n",
        "* We use `run.log()` to record a parameter, num_leaves, and an accuracy measure - the Mean Squared Error (MSE) to the run.  We will be able to review and compare these measures in the Azure Portal at a later time.\n",
        "* We store the resulting model in the **outputs** directory, which is automatically captured by AML when the run is complete.\n",
        "* We use `run.complete()` to indicate that the run is over and results can be captured and finalized"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "## get exact Run from Experiment\n",
        "# from azureml.core import Run\n",
        "# experiment = ws.experiments[\"train-within-notebook-lightgbm\"]\n",
        "# run  = Run(experiment, run_id = '27b20e6d-eb1a-4151-95eb-c94bd55a167d')\n",
        "# run.complete()\n",
        "# run.cancel()\n",
        "# run.fail()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664203929
        },
        "tags": [
          "local run",
          "outputs upload"
        ]
      },
      "outputs": [],
      "source": [
        "from azureml.core import Experiment\n",
        "\n",
        "# Get an experiment object from Azure Machine Learning\n",
        "experiment = Experiment(workspace=ws, name=\"HOL-train-in-notebook-lgbm\")\n",
        "\n",
        "# Create a run object in the experiment\n",
        "run =  experiment.start_logging()\n",
        "\n",
        "\n",
        "# Log the algorithm parameters to the run\n",
        "run.log('num_leaves', 31)\n",
        "run.log('learning_rate', 0.05)\n",
        "run.log('n_estimators', 20)\n",
        "\n",
        "# setup model, train and test\n",
        "gbm = lgb.LGBMRegressor(num_leaves=31,\n",
        "                        learning_rate=0.05,\n",
        "                        n_estimators=20)\n",
        "model_gbm = gbm.fit(x_train, y_train,\n",
        "        eval_set=[(x_test, y_test)],\n",
        "        eval_metric='l1',\n",
        "        early_stopping_rounds=5)\n",
        "\n",
        "preds = model_gbm.predict(x_test)\n",
        "\n",
        "# Output the Mean Squared Error to the notebook and to the run\n",
        "print('Mean Squared Error is', mean_squared_error(y_test, preds))\n",
        "run.log('mse', mean_squared_error(y_test, preds))\n",
        "\n",
        "# Save the model to the outputs directory for capture\n",
        "model_file_name = './outputs/model.pkl'\n",
        "\n",
        "joblib.dump(value = model_gbm, filename = model_file_name)\n",
        "\n",
        "# upload the model file explicitly into artifacts \n",
        "# run.upload_file(name = model_file_name, path_or_stream = model_file_name)\n",
        "\n",
        "# Complete the run\n",
        "run.complete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Simple parameter sweep\n",
        "Now let's take the same concept from above and modify the **num_leaves** parameter.  For each value of num_leaves we will create a run that will store metrics and the resulting model.  In the end we can use the captured run history to determine which model was the best for us to deploy. \n",
        "\n",
        "Note that by using `with experiment.start_logging() as run` AML will automatically call `run.complete()` at the end of each loop.\n",
        "\n",
        "This example also uses the **tqdm** library to provide a thermometer feedback"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664309933
        }
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "from tqdm import tqdm\n",
        "# experiment = Experiment(workspace=ws, name=\"train-locally-within-notebook-sweep3\")\n",
        "# list of numbers from 0 to 1.0 with a 0.05 interval\n",
        "num_leaves_sweep = np.arange(5, 35, 2)\n",
        "mses = []\n",
        "\n",
        "with experiment.start_logging() as run:\n",
        "\n",
        "    for num_leaves in tqdm(num_leaves_sweep):\n",
        "        # create a bunch of runs, each train a model with a different parameters\n",
        "        with run.child_run() as child_run:\n",
        "            gbm = lgb.LGBMRegressor(num_leaves=num_leaves,\n",
        "                                    learning_rate=0.05,\n",
        "                                    n_estimators=20,\n",
        "                                    silent=True)\n",
        "            model_gbm = gbm.fit(x_train, y_train,\n",
        "                    eval_set=[(x_test, y_test)],\n",
        "                    eval_metric='l1',\n",
        "                    early_stopping_rounds=5\n",
        "                    , verbose=False)\n",
        "\n",
        "            preds = model_gbm.predict(x_test)\n",
        "            mse = mean_squared_error(y_true=y_test, y_pred=preds)\n",
        "\n",
        "            # log alpha, mean_squared_error and feature names in run history\n",
        "            child_run.log(name=\"num_leaves\", value=num_leaves)\n",
        "            child_run.log(name=\"mse\", value=mse)\n",
        "            mses.append(mse)\n",
        "\n",
        "    run.log_list(name=\"mses\", value=mses, description='')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Viewing run results\n",
        "Azure Machine Learning stores all the details about the run in the Azure cloud.  Let's access those details by retrieving a link to the run using the default run output.  Clicking on the resulting link will take you to an interactive page presenting all run information."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664310499
        }
      },
      "outputs": [],
      "source": [
        "run"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "An experiment is a logical container in an Azure ML Workspace. It contains a series of trials called Runs. As such, it hosts run records such as run metrics, logs, and other output artifacts from your experiments."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The purpose of this step is to have data points to test the finished model that haven't been used to train the model, in order to measure true accuracy. \n",
        "\n",
        "In other words, a well-trained model should be able to accurately make predictions from data it hasn't already seen. You now have data prepared for auto-training a machine learning model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## PART 3: Train Regresion model on AML remote Compute"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In PART 3, we focus on training models on Remote AML Compute and is divided into two parts where:\n",
        "A) you train single Regreesion model similar to previous section, just using remote compute\n",
        "B) you train multiple Regression models at once and select the best one via Autmated ML componement of Azure Machine Learning service. This happens on remote AML compute - simple auto-scaled cluster of machines for parallel training.\n",
        "\n",
        "The steps are:\n",
        "* configure datasource - remote storage shared between the parallel runs\n",
        "* configure AML compute target\n",
        "* configure and run Automated ML Experiment\n",
        "* review results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Get default blob store associated with your workspace. Alternatively, you can attach your own blob storage to the Workspace - see [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-access-data)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664348851
        },
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "ds = ws.datastores['workspaceblobstore']\n",
        "# ds = ws.get_default_datastore()\n",
        "for attr, value in ds.__dict__.items():\n",
        "    if (attr in ['name', 'datastore_type', 'container_name', 'account_name']):\n",
        "        print(f\"{attr}: {value}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Upload prepared data into associated Datastore."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664371887
        },
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "ds.upload(src_dir='./data', target_path='data', overwrite=True, show_progress=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Configure Compute Target (SDK)\n",
        "\n",
        "Create Compute target in Portal - alternativelly you could create using Pyhton SDK.\n",
        "\n",
        "Reuse the name of the cluster compute you created in preview step and set appropriatelly variable:\n",
        "\n",
        "```python\n",
        "amlcompute_cluster_name = \"<#Name your cluster#>\" "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664381416
        }
      },
      "outputs": [],
      "source": [
        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
        "from azureml.core.compute_target import ComputeTargetException\n",
        "\n",
        "# Choose a name for your CPU cluster\n",
        "amlcompute_cluster_name = \"aml-cluster\" #Name your cluster\n",
        "# amlcompute_cluster_name = \"azdemocluster-f\" #Name your cluster\n",
        "\n",
        "# Verify that cluster does not exist already\n",
        "try:\n",
        "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
        "    print('Found existing cluster, use it.')\n",
        "except ComputeTargetException:\n",
        "    compute_config = AmlCompute.provisioning_configuration(vm_size='Standard_d2_v2', # Standard_F4s_v2\n",
        "                                                           max_nodes=10)\n",
        "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
        "\n",
        "compute_target.wait_for_completion(show_output=True)\n",
        "\n",
        "# Use the 'status' property to get a detailed status for the current cluster. \n",
        "cts = compute_target.status.serialize()\n",
        "print(f'Found existing compute target: {amlcompute_cluster_name}\\n({\"cluster is running\" if (int(cts[\"currentNodeCount\"])>0) else \"cluster is idle\"}) currentNodeCount: {cts[\"currentNodeCount\"]}, vmPriority: {cts[\"vmPriority\"]}, vmSize: {cts[\"vmSize\"]}')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Project folder gets uploaded into docker and will be the ``working directory`` of the executed code."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# from azureml.core.runconfig import DataReferenceConfiguration\n",
        "\n",
        "# dr = DataReferenceConfiguration(datastore_name=ds.name, \n",
        "#                    path_on_datastore='data', \n",
        "#                    path_on_compute='/tmp/azureml_runs',\n",
        "#                    mode='download', # download files from datastore to compute target\n",
        "#                    overwrite=False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664405562
        }
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "project_folder = \"aml_prj\"\n",
        "\n",
        "if not os.path.exists(project_folder):\n",
        "    os.makedirs(project_folder)\n",
        "else:\n",
        "    print(f\"folder '{project_folder}' aready there\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Part 3 A: Train simple Regression model on remote AML Compute"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664408840
        },
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "from azureml.core import Dataset\n",
        "\n",
        "dataset = Dataset.File.from_files((ds, 'data/taxi_final_df_train.csv'))\n",
        "dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664411523
        }
      },
      "outputs": [],
      "source": [
        "from azureml.core import Experiment\n",
        "exp = Experiment(workspace=ws, name=\"HOL-train-on-compute-simple\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664415213
        }
      },
      "outputs": [],
      "source": [
        "from azureml.core import Environment\n",
        "from azureml.core.conda_dependencies import CondaDependencies\n",
        "\n",
        "conda_env = Environment('conda-env')\n",
        "conda_env.python.conda_dependencies = CondaDependencies.create(pip_packages=['azureml-sdk',\n",
        "                                                                             'azureml-dataprep[pandas,fuse]',\n",
        "                                                                             'scikit-learn',\n",
        "                                                                             'lightgbm',\n",
        "                                                                            'joblib'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We must configure the run based on environemnt, script folder with main script and arguments - such as dataset.\n",
        "\n",
        "**Important** the script is just an ordinary `*.py` file located in the script folder named `train.py`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664505989
        }
      },
      "outputs": [],
      "source": [
        "from azureml.core import ScriptRunConfig\n",
        "\n",
        "src = ScriptRunConfig(source_directory=project_folder, \n",
        "                      script='train.py', \n",
        "                      arguments =[dataset.as_named_input('taxi_data').as_mount()])\n",
        "\n",
        "src.run_config.framework = 'python'\n",
        "src.run_config.environment = conda_env\n",
        "src.run_config.target = compute_target.name\n",
        "# src.run_config.data_references = {ds.name: dr}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664516315
        }
      },
      "outputs": [],
      "source": [
        "run = exp.submit(config=src)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      },
      "source": [
        "The experiment is **submitted** to run on remote Compute (AzureML compute cluster) in background. If you run this for the first time, it will run for **10-20min** since it needs to:\n",
        "- build docker image based on your environment\n",
        "- send the image to Azure Container Registry - repository for your environment images\n",
        "- start the compute cluster & upload the docker image in the compute\n",
        "- start the image with the training code\n",
        "\n",
        "Wait until the below widget turns green and says **Completed**."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664545535
        },
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "from azureml.widgets import RunDetails\n",
        "RunDetails(run).show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      },
      "outputs": [],
      "source": [
        "raise Exception('### INTENDED STOP ### to wait for asynchronous training Job')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Part 3B: Trainng multiple models in parallel using Automated ML\n",
        "\n",
        "Observe parameters of DataReferenceConfiguration:\n",
        "\n",
        "* `path_on_datastore`...folder in container\n",
        "* `path_on_compute`...folder where the data is mounted/downloaded\n",
        "* `mode`...wheter download or just mount"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The RunConfiguration sets the docker Python environment - packages and Conda dependencies."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646664970744
        }
      },
      "outputs": [],
      "source": [
        "from azureml.core.runconfig import RunConfiguration\n",
        "from azureml.core.conda_dependencies import CondaDependencies\n",
        "import pkg_resources\n",
        "\n",
        "# create a new RunConfig object\n",
        "conda_run_config = RunConfiguration(framework=\"python\")\n",
        "\n",
        "# Set compute target to the Linux DSVM\n",
        "conda_run_config.target = compute_target\n",
        "# set the data reference of the run coonfiguration\n",
        "# conda_run_config.data_references = {ds.name: dr}\n",
        "\n",
        "pandas_dependency = 'pandas==' + pkg_resources.get_distribution(\"pandas\").version\n",
        "\n",
        "cd = CondaDependencies.create(pip_packages=['azureml-sdk[automl]'], conda_packages=['numpy',pandas_dependency])\n",
        "conda_run_config.environment.python.conda_dependencies = cd"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Automatically train a model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To automatically train a model, take the following steps:\n",
        "1. Define settings for the experiment run. Attach your training data to the configuration, and modify settings that control the training process.\n",
        "1. Submit the experiment for model tuning. After submitting the experiment, the process iterates through different machine learning algorithms and hyperparameter settings, adhering to your defined constraints. It chooses the best-fit model by optimizing an accuracy metric."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Define training settings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Define the experiment parameter and model settings for training. View the full list of [settings](https://docs.microsoft.com/azure/machine-learning/service/how-to-configure-auto-train). Submitting the experiment with these default settings will take approximately 5-10 min, but if you want a shorter run time, reduce the `iterations` parameter.\n",
        "\n",
        "\n",
        "|Property| Value in this tutorial |Description|\n",
        "|----|----|---|\n",
        "|**iteration_timeout_minutes**|2|Time limit in minutes for each iteration. Reduce this value to decrease total runtime.|\n",
        "|**iterations**|20|Number of iterations. In each iteration, a new machine learning model is trained with your data. This is the primary value that affects total run time.|\n",
        "|**primary_metric**| spearman_correlation | Metric that you want to optimize. The best-fit model will be chosen based on this metric.|\n",
        "|**preprocess**| True | By using **True**, the experiment can preprocess the input data (handling missing data, converting text to numeric, etc.)|\n",
        "|**verbosity**| logging.INFO | Controls the level of logging.|\n",
        "|**n_cross_validations**|5|Number of cross-validation splits to perform when validation data is not specified.|\n",
        "|**max_concurrent_iterations**|10|Number of parallel runs - should according to cluster size.|\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646665005005
        }
      },
      "outputs": [],
      "source": [
        "from azureml.train.automl import AutoMLConfig\n",
        "import azureml.dataprep as dprep\n",
        "from azureml.core.dataset import Dataset\n",
        "\n",
        "# train_data_dprep = dprep.auto_read_file(path=ds.path(\"data/taxi_final_df_train.csv\"))\n",
        "# valid_data_dprep = dprep.auto_read_file(path=ds.path(\"data/taxi_final_df_test.csv\"))\n",
        "\n",
        "train_data_dprep = Dataset.Tabular.from_delimited_files(path=(ds, './data/taxi_final_df_test.csv'))\n",
        "valid_data_dprep = Dataset.Tabular.from_delimited_files(path=(ds, './data/taxi_final_df_train.csv'))\n",
        "\n",
        "\n",
        "automl_config = AutoMLConfig(task='regression',\n",
        "                            iteration_timeout_minutes=30,\n",
        "                            iterations=10,\n",
        "                            featurization='auto',\n",
        "                            blocked_models = [\"XGBoostRegressor\",\"ElasticNet\"],\n",
        "                            primary_metric='normalized_root_mean_squared_error',\n",
        "                            training_data=train_data_dprep,\n",
        "                            validation_data=valid_data_dprep,                             \n",
        "                            label_column_name=\"totalAmount\",\n",
        "                            debug_log='automl.log',\n",
        "                            run_configuration=conda_run_config,\n",
        "                            model_explainability=False,\n",
        "                            max_concurrent_iterations=10,\n",
        "                            path= project_folder)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Automated machine learning pre-processing steps (feature normalization, handling missing data, converting text to numeric, etc.) become part of the underlying model. When using the model for predictions, the same pre-processing steps applied during training are applied to your input data automatically."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Train the automatic regression model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Create an experiment object in your workspace. An experiment acts as a container for your individual runs. Pass the defined `automl_config` object to the experiment, and set the output to `True` to view progress during the run. \n",
        "\n",
        "After starting the experiment, the output shown updates live as the experiment runs. For each iteration, you see the model type, the run duration, and the training accuracy. The field `BEST` tracks the best running training score based on your metric type.\n",
        "\n",
        "During the training / experiment running you can observe the result and changes in Azure Portal. Also you can view the state of the runs and results using Widget below in sub-section **Explore results**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646665022758
        }
      },
      "outputs": [],
      "source": [
        "from azureml.core.experiment import Experiment\n",
        "experiment=Experiment(ws, 'HOL-train-automl')\n",
        "remote_run = experiment.submit(automl_config, show_output=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Explore the results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Explore the results of automatic training with a [Jupyter widget](https://docs.microsoft.com/python/api/azureml-widgets/azureml.widgets?view=azure-ml-py). The widget allows you to see a graph and table of all individual run iterations, along with training accuracy metrics and metadata. Additionally, you can filter on different accuracy metrics than your primary metric with the dropdown selector."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646665473561
        }
      },
      "outputs": [],
      "source": [
        "from azureml.widgets import RunDetails\n",
        "RunDetails(remote_run).show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "raise Exception('### INTENDED STOP ### to wait for asynchronous AutoML Job')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Review trained model & results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Retrieve the best model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Select the best model from your iterations. The `get_output` function returns the best run and the fitted model for the last fit invocation. By using the overloads on `get_output`, you can retrieve the best run and fitted model for any logged metric or a particular iteration."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646665510267
        }
      },
      "outputs": [],
      "source": [
        "best_run, fitted_model = remote_run.get_output()\n",
        "print(\"Best Run:\")\n",
        "print(best_run)\n",
        "\n",
        "print(\"\")\n",
        "print(\"Fitted model:\")\n",
        "print(fitted_model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Test the best model accuracy\n",
        "\n",
        "Use the best model to run predictions on the test data set to predict taxi fares. The function `predict` uses the best model and predicts the values of y, **trip cost**, from the `x_test` data set. Print the first 10 predicted cost values from `y_predict`.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646665531998
        }
      },
      "outputs": [],
      "source": [
        "test = pd.read_csv(\"./data/taxi_final_df_test.csv\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646665536848
        }
      },
      "outputs": [],
      "source": [
        "test.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646665541447
        }
      },
      "outputs": [],
      "source": [
        "y_test = test.pop(\"totalAmount\")\n",
        "x_test = test"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646665548038
        }
      },
      "outputs": [],
      "source": [
        "y_predict = fitted_model.predict(x_test)\n",
        "print(y_predict[:10])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Calculate the `root mean squared error` of the results. Convert the `y_test` dataframe to a list to compare to the predicted values. The function `mean_squared_error` takes two arrays of values and calculates the average squared error between them. Taking the square root of the result gives an error in the same units as the y variable, **cost**. It indicates roughly how far the taxi fare predictions are from the actual fares.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646665569352
        }
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import mean_squared_error\n",
        "from math import sqrt\n",
        "\n",
        "y_actual = y_test.values.flatten().tolist()\n",
        "rmse = sqrt(mean_squared_error(y_actual, y_predict))\n",
        "print(f\"RMSE: {rmse}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Run the following code to calculate mean absolute percent error (MAPE) by using the full `y_actual` and `y_predict` data sets. This metric calculates an absolute difference between each predicted and actual value and sums all the differences. Then it expresses that sum as a percent of the total of the actual values.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646665614477
        }
      },
      "outputs": [],
      "source": [
        "sum_actuals = sum_errors = 0\n",
        "\n",
        "for actual_val, predict_val in zip(y_actual, y_predict):\n",
        "    abs_error = actual_val - predict_val\n",
        "    if abs_error < 0:\n",
        "        abs_error = abs_error * -1\n",
        "\n",
        "    sum_errors = sum_errors + abs_error\n",
        "    sum_actuals = sum_actuals + actual_val\n",
        "\n",
        "mean_abs_percent_error = sum_errors / sum_actuals\n",
        "print(\"Model MAPE:\")\n",
        "print(mean_abs_percent_error)\n",
        "print()\n",
        "print(\"Model Accuracy:\")\n",
        "print(1 - mean_abs_percent_error)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "From the two prediction accuracy metrics, you see that the model is fairly good at predicting taxi fares from the data set's features, typically within +- $4.00, and approximately 15% error. \n",
        "\n",
        "The traditional machine learning model development process is highly resource-intensive, and requires significant domain knowledge and time investment to run and compare the results of dozens of models. Using automated machine learning is a great way to rapidly test many different models for your scenario."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646665582782
        }
      },
      "outputs": [],
      "source": [
        "%matplotlib inline\n",
        "import matplotlib.pyplot as plt\n",
        "test_pred = plt.scatter(y_actual, y_predict, color='b')\n",
        "test_test = plt.scatter(y_actual, y_actual, color='g')\n",
        "plt.legend((test_pred, test_test), ('prediction', 'truth'), loc='upper left', fontsize=8)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## PART 4: Model Deployment [optional]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In this part of the lab, you use trained best model to create deployment as webservice through Managed real-time Endpoint.\n",
        "\n",
        "You will move for a while from this notebook to AzureML studio (just to showcase), but you could do similar through Python SDK or CLI. The steps of deployment process are:\n",
        "\n",
        "* Navigating to experiment for AutoML and select the best model\n",
        "* Deploy the model to real-time endpoint\n",
        "* test the endpoint with your data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "First, navigate to you Experiments and find the AutoML Experiment.\n",
        "\n",
        "![deployment1](./media/deployment01.jpg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Then, on the Experiment, locate your latest run a select Models to see all trained model during AutoML:\n",
        "![deploymen2](./media/deployment02.jpg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "On th particular model, select Deploy -> Deploy to real-time endpoint (Preview) and follow the wizard.\n",
        "![deployment3](./media/deployment03.jpg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In the wizard, give your enpoint unique name (it is name of the publicly available service), and leave next steps in deafult values.\n",
        "![deployment4](./media/deployment04.jpg)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "At the Compute stage of the wizard lower number of instances to `1`:\n",
        "![deployment5](./media/deployment05.jpg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Finish the deployment by hit on blue \"Create\" button on th mottom of the page.\n",
        "![deployment6](./media/deployment06.jpg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "After succesfull deployment, navigate to your Endpoints and select newly deployed real-time Endpoint to see the details.\n",
        "![deployment7](./media/deployment07.jpg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can navigate to **Consume** tab, to see the Key, Scoring URI and example in various programming languages on how to call the enpoint -> choose Python and copy & paste into cell below (or only fill the necessary parameters).\n",
        "![deployment8](./media/deployment08.jpg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Test the webservices - score in real-time"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1646682226196
        },
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      },
      "outputs": [],
      "source": [
        "service_name =\"***\" # YOUR ENDPONT NAME\n",
        "scoring_uri = \"***\" # YOUR SCORING URI\n",
        "scoring_key = \"***\" # YOUR SCORING KEY \n",
        "\n",
        "import requests\n",
        "import json\n",
        "\n",
        "# get sample test/validation data\n",
        "sample_df = pd.read_csv(\"./data/taxi_final_df_test.csv\").sample()\n",
        "vals = sample_df[[\"vendorID\",\"passengerCount\",\"tripDistance\",\"month_num\",\"day_of_month\",\"day_of_week\",\"hour_of_day\",\"totalAmount\"]].values\n",
        "sample_data = vals.tolist()[0][0:-1]\n",
        "sample_target = vals.tolist()[0][-1]\n",
        "test_sample = json.dumps({\"Inputs\": {\"data\": sample_data} })\n",
        "\n",
        "# Convert to JSON string\n",
        "input_data = json.dumps(test_sample)\n",
        "\n",
        "# Set the content type\n",
        "headers = {'Content-Type': 'application/json'}\n",
        "# If authentication is enabled, set the authorization header\n",
        "headers['Authorization'] = f'Bearer {scoring_key}'\n",
        "\n",
        "# Make the request and display the response\n",
        "resp = requests.post(scoring_uri, input_data, headers=headers)\n",
        "# print(resp.text)\n",
        "\n",
        "resp_json = json.loads(resp.text)\n",
        "prediction = resp_json[\"Results\"][0]\n",
        "\n",
        "print(f\"Prediction: {prediction}\\nActuals: {sample_target}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Clean up resources"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Do not complete this section if you plan on running other Azure Machine Learning service tutorials."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Delete service"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "service.delete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Stop the notebook VM"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If you used a cloud notebook server, stop the VM when you are not using it to reduce cost."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "1. In your workspace, select **Notebook VMs**.\n",
        "1. From the list, select the VM.\n",
        "1. Select **Stop**.\n",
        "1. When you're ready to use the server again, select **Start**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Delete everything"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If you don't plan to use the resources you created, delete them, so you don't incur any charges."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "1. In the Azure portal, select **Resource groups** on the far left.\n",
        "1. From the list, select the resource group you created.\n",
        "1. Select **Delete resource group**.\n",
        "1. Enter the resource group name. Then select **Delete**.\n",
        "\n",
        "You can also keep the resource group but delete a single workspace. Display the workspace properties and select **Delete**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Next steps"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In this machine learning lab, you did the following tasks:\n",
        "\n",
        "> * Configured a workspace and prepared data for an experiment.\n",
        "> * Trained by using aregression model locally &  with custom parameters.\n",
        "> * Trained by using autmated ML aregression model on achine learning compute.\n",
        "> * Explored and reviewed training results.\n",
        "> * Deploy model to ACI and test the web service\n",
        "\n",
        "Visit [docs](https://docs.microsoft.com/azure/machine-learning/service/) with Azure Machine Learning service documenation and tutorials.\n",
        "\n",
        "Learn by examples and code at [AML GitHub](https://github.com/Azure/MachineLearningNotebooks)"
      ]
    }
  ],
  "metadata": {
    "authors": [
      {
        "name": "jeffshep"
      }
    ],
    "categories": [
      "tutorials"
    ],
    "kernel_info": {
      "name": "python3-azureml"
    },
    "kernelspec": {
      "display_name": "Python 3.6 - AzureML",
      "language": "python",
      "name": "python3-azureml"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.9"
    },
    "microsoft": {
      "host": {
        "AzureML": {
          "notebookHasBeenCompleted": true
        }
      }
    },
    "msauthor": "trbye",
    "nteract": {
      "version": "nteract-front-end@1.0.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}