tutorials/image-classification-mnist-data/img-classification-part3-deploy-encrypted.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
        "\n",
        "Licensed under the MIT License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Tutorial #3:  Deploy an image classification model for encrypted inferencing in Azure Container Instance (ACI)\n",
        "\n",
        "This tutorial is **a new addition to the two-part series**. In the [previous tutorial](img-classification-part1-training.ipynb), you trained machine learning models and then registered a model in your workspace on the cloud.  \n",
        "\n",
        "Now, you're ready to deploy the model as a encrypted inferencing web service in [Azure Container Instances](https://docs.microsoft.com/azure/container-instances/) (ACI). A web service is an image, in this case a Docker image, that encapsulates the scoring logic and the model itself. \n",
        "\n",
        "In this part of the tutorial, you use Azure Machine Learning service (Preview) to:\n",
        "\n",
        "> * Set up your testing environment\n",
        "> * Retrieve the model from your workspace\n",
        "> * Test the model locally\n",
        "> * Deploy the model to ACI\n",
        "> * Test the deployed model\n",
        "\n",
        "ACI is a great solution for testing and understanding the workflow. For scalable production deployments, consider using Azure Kubernetes Service. For more information, see [how to deploy and where](https://docs.microsoft.com/azure/machine-learning/service/how-to-deploy-and-where).\n",
        "\n",
        "\n",
        "## Prerequisites\n",
        "\n",
        "Complete the model training in the [Tutorial #1: Train an image classification model with Azure Machine Learning](train-models.ipynb) notebook.  \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# If you did NOT complete the tutorial, you can instead run this cell \n",
        "# This will register a model and download the data needed for this tutorial\n",
        "# These prerequisites are created in the training tutorial\n",
        "# Feel free to skip this cell if you completed the training tutorial \n",
        "\n",
        "# register a model\n",
        "from azureml.core import Workspace\n",
        "ws = Workspace.from_config()\n",
        "\n",
        "from azureml.core.model import Model\n",
        "\n",
        "model_name = \"sklearn_mnist\"\n",
        "model = Model.register(model_path=\"sklearn_mnist_model.pkl\",\n",
        "                        model_name=model_name,\n",
        "                        tags={\"data\": \"mnist\", \"model\": \"classification\"},\n",
        "                        description=\"Mnist handwriting recognition\",\n",
        "                        workspace=ws)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Setup the Environment \n",
        "\n",
        "Add `encrypted-inference` package as a conda dependency "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.environment import Environment\n",
        "from azureml.core.conda_dependencies import CondaDependencies\n",
        "\n",
        "# to install required packages\n",
        "env = Environment('tutorial-env')\n",
        "cd = CondaDependencies.create(pip_packages=['azureml-dataprep[pandas,fuse]>=1.1.14', 'azureml-defaults', 'azure-storage-blob', 'encrypted-inference==0.9'], conda_packages = ['scikit-learn==0.22.1'])\n",
        "\n",
        "env.python.conda_dependencies = cd\n",
        "\n",
        "# Register environment to re-use later\n",
        "env.register(workspace = ws)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Set up the environment\n",
        "\n",
        "Start by setting up a testing environment.\n",
        "\n",
        "### Import packages\n",
        "\n",
        "Import the Python packages needed for this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "check version"
        ]
      },
      "outputs": [],
      "source": [
        "%matplotlib inline\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        " \n",
        "import azureml.core\n",
        "\n",
        "# display the core SDK version number\n",
        "print(\"Azure ML SDK Version: \", azureml.core.VERSION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Install Homomorphic Encryption based library for Secure Inferencing\n",
        "\n",
        "Our library is based on [Microsoft SEAL](https://github.com/Microsoft/SEAL) and pubished to [PyPi.org](https://pypi.org/project/encrypted-inference) as an easy to use package "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "!pip install encrypted-inference==0.9"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Deploy as web service\n",
        "\n",
        "Deploy the model as a web service hosted in ACI. \n",
        "\n",
        "To build the correct environment for ACI, provide the following:\n",
        "* A scoring script to show how to use the model\n",
        "* A configuration file to build the ACI\n",
        "* The model you trained before\n",
        "\n",
        "### Create scoring script\n",
        "\n",
        "Create the scoring script, called score.py, used by the web service call to show how to use the model.\n",
        "\n",
        "You must include two required functions into the scoring script:\n",
        "* The `init()` function, which typically loads the model into a global object. This function is run only once when the Docker container is started. \n",
        "\n",
        "* The `run(input_data)` function uses the model to predict a value based on the input data. Inputs and outputs to the run typically use JSON for serialization and de-serialization, but other formats are supported. The function fetches homomorphic encryption based public keys that are uploaded by the service caller. \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%writefile score.py\n",
        "import json\n",
        "import os\n",
        "import pickle\n",
        "import joblib\n",
        "from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient, PublicAccess\n",
        "from encrypted.inference.eiserver import EIServer\n",
        "\n",
        "def init():\n",
        "    global model\n",
        "    # AZUREML_MODEL_DIR is an environment variable created during deployment.\n",
        "    # It is the path to the model folder (./azureml-models/$MODEL_NAME/$VERSION)\n",
        "    # For multiple models, it points to the folder containing all deployed models (./azureml-models)\n",
        "    model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), 'sklearn_mnist_model.pkl')\n",
        "    model = joblib.load(model_path)\n",
        "\n",
        "    global server\n",
        "    server = EIServer(model.coef_, model.intercept_, verbose=True)\n",
        "\n",
        "def run(raw_data):\n",
        "\n",
        "    json_properties = json.loads(raw_data)\n",
        "\n",
        "    key_id = json_properties['key_id']\n",
        "    conn_str = json_properties['conn_str']\n",
        "    container = json_properties['container']\n",
        "    data = json_properties['data']\n",
        "\n",
        "    # download the Galois keys from blob storage\n",
        "    #TODO optimize by caching the keys locally  \n",
        "    blob_service_client = BlobServiceClient.from_connection_string(conn_str=conn_str)\n",
        "    blob_client = blob_service_client.get_blob_client(container=container, blob=key_id)\n",
        "    public_keys = blob_client.download_blob().readall()\n",
        "    \n",
        "    result = {}\n",
        "    # make prediction\n",
        "    result = server.predict(data, public_keys)\n",
        "\n",
        "    # you can return any data type as long as it is JSON-serializable\n",
        "    return result"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Create configuration file\n",
        "\n",
        "Create a deployment configuration file and specify the number of CPUs and gigabyte of RAM needed for your ACI container. While it depends on your model, the default of 1 core and 1 gigabyte of RAM is usually sufficient for many models. If you feel you need more later, you would have to recreate the image and redeploy the service."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "configure web service",
          "aci"
        ]
      },
      "outputs": [],
      "source": [
        "from azureml.core.webservice import AciWebservice\n",
        "\n",
        "aciconfig = AciWebservice.deploy_configuration(cpu_cores=1, \n",
        "                                               memory_gb=1, \n",
        "                                               tags={\"data\": \"MNIST\",  \"method\" : \"sklearn\"}, \n",
        "                                               description='Encrypted Predict MNIST with sklearn + SEAL')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Deploy in ACI\n",
        "Estimated time to complete: **about 2-5 minutes**\n",
        "\n",
        "Configure the image and deploy. The following code goes through these steps:\n",
        "\n",
        "1. Create environment object containing dependencies needed by the model using the environment file (`myenv.yml`)\n",
        "1. Create inference configuration necessary to deploy the model as a web service using:\n",
        "   * The scoring file (`score.py`)\n",
        "   * envrionment object created in previous step\n",
        "1. Deploy the model to the ACI container.\n",
        "1. Get the web service HTTP endpoint."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "configure image",
          "create image",
          "deploy web service",
          "aci"
        ]
      },
      "outputs": [],
      "source": [
        "%%time\n",
        "import uuid\n",
        "from azureml.core.webservice import Webservice\n",
        "from azureml.core.model import InferenceConfig\n",
        "from azureml.core.environment import Environment\n",
        "from azureml.core import Workspace\n",
        "from azureml.core.model import Model\n",
        "\n",
        "ws = Workspace.from_config()\n",
        "model = Model(ws, 'sklearn_mnist')\n",
        "\n",
        "myenv = Environment.get(workspace=ws, name=\"tutorial-env\")\n",
        "inference_config = InferenceConfig(entry_script=\"score.py\", environment=myenv)\n",
        "\n",
        "service_name = 'sklearn-mnist-svc-' + str(uuid.uuid4())[:4]\n",
        "service = Model.deploy(workspace=ws, \n",
        "                       name=service_name, \n",
        "                       models=[model], \n",
        "                       inference_config=inference_config, \n",
        "                       deployment_config=aciconfig)\n",
        "\n",
        "service.wait_for_deployment(show_output=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Get the scoring web service's HTTP endpoint, which accepts REST client calls. This endpoint can be shared with anyone who wants to test the web service or integrate it into an application."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "get scoring uri"
        ]
      },
      "outputs": [],
      "source": [
        "print(service.scoring_uri)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Test the model\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Download test data\n",
        "Download the test data to the **./data/** directory"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import os\n",
        "from azureml.core import Dataset\n",
        "from azureml.opendatasets import MNIST\n",
        "\n",
        "data_folder = os.path.join(os.getcwd(), 'data')\n",
        "os.makedirs(data_folder, exist_ok=True)\n",
        "\n",
        "mnist_file_dataset = MNIST.get_file_dataset()\n",
        "mnist_file_dataset.download(data_folder, overwrite=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Load test data\n",
        "\n",
        "Load the test data from the **./data/** directory created during the training tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from utils import load_data\n",
        "import os\n",
        "import glob\n",
        "\n",
        "data_folder = os.path.join(os.getcwd(), 'data')\n",
        "# note we also shrink the intensity values (X) from 0-255 to 0-1. This helps the neural network converge faster\n",
        "X_test = load_data(glob.glob(os.path.join(data_folder,\"**/t10k-images-idx3-ubyte.gz\"), recursive=True)[0], False) / 255.0\n",
        "y_test = load_data(glob.glob(os.path.join(data_folder,\"**/t10k-labels-idx1-ubyte.gz\"), recursive=True)[0], True).reshape(-1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Predict test data\n",
        "\n",
        "Feed the test dataset to the model to get predictions.\n",
        "\n",
        "\n",
        "The following code goes through these steps:\n",
        "\n",
        "1. Create our Homomorphic Encryption based client \n",
        "\n",
        "1. Upload HE generated public keys \n",
        "\n",
        "1. Encrypt the data\n",
        "\n",
        "1. Send the data as JSON to the web service hosted in ACI. \n",
        "\n",
        "1. Use the SDK's `run` API to invoke the service. You can also make raw calls using any HTTP tool such as curl."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Create our Homomorphic Encryption based client \n",
        "\n",
        "Create a new EILinearRegressionClient and setup the public keys "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from encrypted.inference.eiclient import EILinearRegressionClient\n",
        "\n",
        "# Create a new Encrypted inference client and a new secret key.\n",
        "edp = EILinearRegressionClient(verbose=True)\n",
        "\n",
        "public_keys_blob, public_keys_data = edp.get_public_keys()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Upload HE generated public keys\n",
        "\n",
        "Upload the public keys to the workspace default blob store. This will allow us to share the keys with the inference server"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import azureml.core\n",
        "from azureml.core import Workspace, Datastore\n",
        "import os\n",
        "\n",
        "ws = Workspace.from_config()\n",
        "\n",
        "datastore = ws.get_default_datastore()\n",
        "container_name=datastore.container_name\n",
        "\n",
        "# Create a local file and write the keys to it\n",
        "public_keys = open(public_keys_blob, \"wb\")\n",
        "public_keys.write(public_keys_data)\n",
        "public_keys.close()\n",
        "\n",
        "# Upload the file to blob store\n",
        "datastore.upload_files([public_keys_blob])\n",
        "\n",
        "# Delete the local file\n",
        "os.remove(public_keys_blob)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Encrypt the data "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "#choose any one sample from the test data \n",
        "sample_index = 1\n",
        "\n",
        "#encrypt the data\n",
        "raw_data = edp.encrypt(X_test[sample_index])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Send the test data to the webservice hosted in ACI\n",
        "\n",
        "Feed the test dataset to the model to get predictions. We will need to send the connection string to the blob storage where the public keys were uploaded \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import json\n",
        "from azureml.core import Webservice\n",
        "\n",
        "service = Webservice(ws, service_name)\n",
        "\n",
        "#pass the connection string for blob storage to give the server access to the uploaded public keys \n",
        "conn_str_template = 'DefaultEndpointsProtocol={};AccountName={};AccountKey={};EndpointSuffix=core.windows.net'\n",
        "conn_str = conn_str_template.format(datastore.protocol, datastore.account_name, datastore.account_key)\n",
        "\n",
        "#build the json \n",
        "data = json.dumps({\"data\": raw_data, \"key_id\" : public_keys_blob, \"conn_str\" : conn_str, \"container\" : container_name })\n",
        "data = bytes(data, encoding='ASCII')\n",
        "\n",
        "print ('Making an encrypted inference web service call ')\n",
        "eresult = service.run(input_data=data)\n",
        "\n",
        "print ('Received encrypted inference results')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Decrypt the data\n",
        "\n",
        "Use the client to decrypt the results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import numpy as np \n",
        "\n",
        "results = edp.decrypt(eresult)\n",
        "\n",
        "print ('Decrypted the results ', results)\n",
        "\n",
        "#Apply argmax to identify the prediction result\n",
        "prediction = np.argmax(results)\n",
        "\n",
        "print ( ' Prediction : ', prediction)\n",
        "print ( ' Actual Label : ', y_test[sample_index])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Clean up resources\n",
        "\n",
        "To keep the resource group and workspace for other tutorials and exploration, you can delete only the ACI deployment using this API call:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "delete web service"
        ]
      },
      "outputs": [],
      "source": [
        "service.delete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "If you're not going to use what you've created here, delete the resources you just created with this quickstart so you don't incur any charges. In the Azure portal, select and delete your resource group. You can also keep the resource group, but delete a single workspace by displaying the workspace properties and selecting the Delete button.\n",
        "\n",
        "\n",
        "## Next steps\n",
        "\n",
        "In this Azure Machine Learning tutorial, you used Python to:\n",
        "\n",
        "> * Set up your testing environment\n",
        "> * Retrieve the model from your workspace\n",
        "> * Test the model locally\n",
        "> * Deploy the model to ACI\n",
        "> * Test the deployed model\n",
        " \n",
        "You can also try out the [regression tutorial](regression-part1-data-prep.ipynb)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/tutorials/img-classification-part2-deploy.png)"
      ]
    }
  ],
  "metadata": {
    "authors": [
      {
        "name": "vkanne"
      }
    ],
    "celltoolbar": "Edit Metadata",
    "kernelspec": {
      "display_name": "Python 3.6",
      "language": "python",
      "name": "python36"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6"
    },
    "msauthor": "vkanne"
  },
  "nbformat": 4,
  "nbformat_minor": 2
}