tutorials_zh/05._Implement_a_Simple_AutoML_System.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## 05. 实现一个简单的AutoML系统"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "### AutoML系统的顶层设计"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "我们在上个教程中，学习到了UltraOpt的设计哲学是将优化器与评价器分离，如图所示:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"668pt\" height=\"79pt\"\n",
       " viewBox=\"0.00 0.00 668.18 78.70\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 74.6978)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-74.6978 664.18,-74.6978 664.18,4 -4,4\"/>\n",
       "<!-- config space -->\n",
       "<g id=\"node1\" class=\"node\"><title>config space</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"55.2451\" cy=\"-38.6978\" rx=\"55.4913\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"55.2451\" y=\"-34.9978\" font-family=\"Times,serif\" font-size=\"14.00\">config space</text>\n",
       "</g>\n",
       "<!-- optimizer -->\n",
       "<g id=\"node2\" class=\"node\"><title>optimizer</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"263.49,-56.6978 194.49,-56.6978 194.49,-20.6978 263.49,-20.6978 263.49,-56.6978\"/>\n",
       "<text text-anchor=\"middle\" x=\"228.99\" y=\"-34.9978\" font-family=\"Times,serif\" font-size=\"14.00\">optimizer</text>\n",
       "</g>\n",
       "<!-- config space&#45;&gt;optimizer -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>config space&#45;&gt;optimizer</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M110.586,-38.6978C134.341,-38.6978 161.834,-38.6978 184.24,-38.6978\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"184.468,-42.1979 194.468,-38.6978 184.468,-35.1979 184.468,-42.1979\"/>\n",
       "<text text-anchor=\"middle\" x=\"152.49\" y=\"-42.4978\" font-family=\"Times,serif\" font-size=\"14.00\">initialize</text>\n",
       "</g>\n",
       "<!-- config -->\n",
       "<g id=\"node3\" class=\"node\"><title>config</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"352.637\" cy=\"-51.6978\" rx=\"33.2948\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"352.637\" y=\"-47.9978\" font-family=\"Times,serif\" font-size=\"14.00\">config</text>\n",
       "</g>\n",
       "<!-- optimizer&#45;&gt;config -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>optimizer&#45;&gt;config</title>\n",
       "<path fill=\"none\" stroke=\"blue\" d=\"M263.725,-50.7587C269.578,-52.3944 275.659,-53.8128 281.49,-54.6978 290.597,-56.0798 300.439,-56.3846 309.789,-56.1277\"/>\n",
       "<polygon fill=\"blue\" stroke=\"blue\" points=\"310.253,-59.6093 320.071,-55.6264 309.912,-52.6176 310.253,-59.6093\"/>\n",
       "<text text-anchor=\"start\" x=\"281.49\" y=\"-60.4978\" font-family=\"Times,serif\" font-weight=\"bold\" font-size=\"14.00\">ask</text>\n",
       "</g>\n",
       "<!-- config&#45;&gt;optimizer -->\n",
       "<g id=\"edge5\" class=\"edge\"><title>config&#45;&gt;optimizer</title>\n",
       "<path fill=\"none\" stroke=\"red\" d=\"M325.2,-41.5843C317.654,-39.1603 309.346,-36.9207 301.49,-35.6978 292.571,-34.3094 282.955,-33.9507 273.756,-34.1491\"/>\n",
       "<polygon fill=\"red\" stroke=\"red\" points=\"273.448,-30.659 263.608,-34.5846 273.749,-37.6525 273.448,-30.659\"/>\n",
       "<text text-anchor=\"start\" x=\"281.99\" y=\"-40.4978\" font-family=\"Times,serif\" font-weight=\"bold\" font-size=\"14.00\">tell</text>\n",
       "</g>\n",
       "<!-- evaluator -->\n",
       "<g id=\"node5\" class=\"node\"><title>evaluator</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"528.784,-69.6978 461.784,-69.6978 461.784,-33.6978 528.784,-33.6978 528.784,-69.6978\"/>\n",
       "<text text-anchor=\"middle\" x=\"495.284\" y=\"-47.9978\" font-family=\"Times,serif\" font-size=\"14.00\">evaluator</text>\n",
       "</g>\n",
       "<!-- config&#45;&gt;evaluator -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>config&#45;&gt;evaluator</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M385.905,-51.6978C405.42,-51.6978 430.53,-51.6978 451.666,-51.6978\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"451.68,-55.1979 461.68,-51.6978 451.68,-48.1979 451.68,-55.1979\"/>\n",
       "<text text-anchor=\"middle\" x=\"423.784\" y=\"-55.4978\" font-family=\"Times,serif\" font-size=\"14.00\">send to</text>\n",
       "</g>\n",
       "<!-- loss -->\n",
       "<g id=\"node4\" class=\"node\"><title>loss</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"635.482\" cy=\"-24.6978\" rx=\"24.8972\" ry=\"24.8972\"/>\n",
       "<text text-anchor=\"middle\" x=\"635.482\" y=\"-20.9978\" font-family=\"Times,serif\" font-size=\"14.00\">loss</text>\n",
       "</g>\n",
       "<!-- loss&#45;&gt;optimizer -->\n",
       "<g id=\"edge6\" class=\"edge\"><title>loss&#45;&gt;optimizer</title>\n",
       "<path fill=\"none\" stroke=\"red\" d=\"M610.739,-23.6398C558.186,-21.53 428.083,-17.5842 319.49,-24.6978 304.589,-25.674 288.429,-27.852 273.993,-30.2061\"/>\n",
       "<polygon fill=\"red\" stroke=\"red\" points=\"273.101,-26.8077 263.83,-31.9367 274.276,-33.7084 273.101,-26.8077\"/>\n",
       "<text text-anchor=\"start\" x=\"414.284\" y=\"-25.4978\" font-family=\"Times,serif\" font-weight=\"bold\" font-size=\"14.00\">tell</text>\n",
       "</g>\n",
       "<!-- evaluator&#45;&gt;loss -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>evaluator&#45;&gt;loss</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M529.035,-45.2992C550.809,-41.0451 579.315,-35.4759 601.198,-31.2006\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"601.9,-34.6296 611.044,-29.277 600.558,-27.7595 601.9,-34.6296\"/>\n",
       "<text text-anchor=\"middle\" x=\"569.784\" y=\"-44.4978\" font-family=\"Times,serif\" font-size=\"14.00\">evaluate</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f0ff0956160>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from graphviz import Digraph; g = Digraph()\n",
    "g.node(\"config space\", shape=\"ellipse\"); g.node(\"optimizer\", shape=\"box\")\n",
    "g.node(\"config\", shape=\"ellipse\"); g.node(\"loss\", shape=\"circle\"); g.node(\"evaluator\", shape=\"box\")\n",
    "g.edge(\"config space\", \"optimizer\", label=\"initialize\"); g.edge(\"optimizer\", \"config\", label=\"<<b>ask</b>>\", color='blue')\n",
    "g.edge(\"config\",\"evaluator\" , label=\"send to\"); g.edge(\"evaluator\",\"loss\" , label=\"evaluate\")\n",
    "g.edge(\"config\", \"optimizer\", label=\"<<b>tell</b>>\", color='red'); g.edge(\"loss\", \"optimizer\", label=\"<<b>tell</b>>\", color='red')\n",
    "g.graph_attr['rankdir'] = 'LR'; g"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "而当我们要解决AutoML问题时，我们可以这样定义AutoML系统结构："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "![AutoML Structure](https://img-blog.csdnimg.cn/20201225225241790.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "### 用HDL定义一个简单的AutoML配置空间"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "在`03. Conditional Parameter`教程中，我们知道了AutoML问题的优化可以视为一个CASH问题，不仅涉及算法选择，还涉及超参优化。我们用HDL定义一个简单的CASH问题的配置空间："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "from ultraopt.hdl import hdl2cs, plot_hdl, layering_config, plot_layered_dict\n",
    "HDL = {\n",
    "    'classifier(choice)':{\n",
    "        \"LinearSVC\": {\n",
    "          \"max_iter\": {\"_type\": \"int_quniform\",\"_value\": [300, 3000, 100], \"_default\": 600},\n",
    "          \"penalty\":  {\"_type\": \"choice\", \"_value\": [\"l1\", \"l2\"],\"_default\": \"l2\"},\n",
    "          \"dual\": {\"_type\": \"choice\", \"_value\": [True, False],\"_default\": False},\n",
    "          \"loss\":  {\"_type\": \"choice\", \"_value\": [\"hinge\", \"squared_hinge\"],\"_default\": \"squared_hinge\"},\n",
    "          \"C\": {\"_type\": \"loguniform\", \"_value\": [0.01, 10000],\"_default\": 1.0},\n",
    "          \"multi_class\": \"ovr\",\n",
    "          \"random_state\": 42,\n",
    "          \"__forbidden\": [\n",
    "              {\"penalty\": \"l1\",\"loss\": \"hinge\"},\n",
    "              {\"penalty\": \"l2\",\"dual\": False,\"loss\": \"hinge\"},\n",
    "              {\"penalty\": \"l1\",\"dual\": False},\n",
    "              {\"penalty\": \"l1\",\"dual\": True,\"loss\": \"squared_hinge\"},\n",
    "          ]\n",
    "        },\n",
    "        \"RandomForestClassifier\": {\n",
    "          \"n_estimators\": {\"_type\": \"int_quniform\",\"_value\": [10, 200, 10], \"_default\": 100},\n",
    "          \"criterion\": {\"_type\": \"choice\",\"_value\": [\"gini\", \"entropy\"],\"_default\": \"gini\"},\n",
    "          \"max_features\": {\"_type\": \"choice\",\"_value\": [\"sqrt\",\"log2\"],\"_default\": \"sqrt\"},\n",
    "          \"min_samples_split\": {\"_type\": \"int_uniform\", \"_value\": [2, 20],\"_default\": 2},\n",
    "          \"min_samples_leaf\": {\"_type\": \"int_uniform\", \"_value\": [1, 20],\"_default\": 1},\n",
    "          \"bootstrap\": {\"_type\": \"choice\",\"_value\": [True, False],\"_default\": True},\n",
    "          \"random_state\": 42\n",
    "        },\n",
    "        \"KNeighborsClassifier\": {\n",
    "          \"n_neighbors\": {\"_type\": \"int_loguniform\", \"_value\": [1,100],\"_default\": 3},\n",
    "          \"weights\" : {\"_type\": \"choice\", \"_value\": [\"uniform\", \"distance\"],\"_default\": \"uniform\"},\n",
    "          \"p\": {\"_type\": \"choice\", \"_value\": [1, 2],\"_default\": 2},\n",
    "        },\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n",
       " -->\n",
       "<!-- Title: Config Space Pages: 1 -->\n",
       "<svg width=\"1080pt\" height=\"116pt\"\n",
       " viewBox=\"0.00 0.00 1080.00 115.61\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(0.614948 0.614948) rotate(0) translate(4 184)\">\n",
       "<title>Config Space</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-184 1752.25,-184 1752.25,4 -4,4\"/>\n",
       "<!-- classifier -->\n",
       "<g id=\"node1\" class=\"node\"><title>classifier</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"999.811,-162 975.529,-180 926.964,-180 902.682,-162 926.964,-144 975.529,-144 999.811,-162\"/>\n",
       "<text text-anchor=\"middle\" x=\"951.246\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\">classifier</text>\n",
       "</g>\n",
       "<!-- LinearSVC -->\n",
       "<g id=\"node2\" class=\"node\"><title>LinearSVC</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"367.246,-108 289.246,-108 289.246,-72 367.246,-72 367.246,-108\"/>\n",
       "<text text-anchor=\"middle\" x=\"328.246\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">LinearSVC</text>\n",
       "</g>\n",
       "<!-- classifier&#45;&gt;LinearSVC -->\n",
       "<g id=\"edge8\" class=\"edge\"><title>classifier&#45;&gt;LinearSVC</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M909.887,-156.353C799.242,-143.921 496.782,-109.937 377.526,-96.537\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"377.903,-93.0575 367.575,-95.4189 377.121,-100.014 377.903,-93.0575\"/>\n",
       "</g>\n",
       "<!-- RandomForestClassifier -->\n",
       "<g id=\"node10\" class=\"node\"><title>RandomForestClassifier</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1025.75,-108 876.746,-108 876.746,-72 1025.75,-72 1025.75,-108\"/>\n",
       "<text text-anchor=\"middle\" x=\"951.246\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">RandomForestClassifier</text>\n",
       "</g>\n",
       "<!-- classifier&#45;&gt;RandomForestClassifier -->\n",
       "<g id=\"edge16\" class=\"edge\"><title>classifier&#45;&gt;RandomForestClassifier</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M951.246,-143.697C951.246,-135.983 951.246,-126.712 951.246,-118.112\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"954.746,-118.104 951.246,-108.104 947.746,-118.104 954.746,-118.104\"/>\n",
       "</g>\n",
       "<!-- KNeighborsClassifier -->\n",
       "<g id=\"node17\" class=\"node\"><title>KNeighborsClassifier</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1650.25,-108 1516.25,-108 1516.25,-72 1650.25,-72 1650.25,-108\"/>\n",
       "<text text-anchor=\"middle\" x=\"1583.25\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">KNeighborsClassifier</text>\n",
       "</g>\n",
       "<!-- classifier&#45;&gt;KNeighborsClassifier -->\n",
       "<g id=\"edge20\" class=\"edge\"><title>classifier&#45;&gt;KNeighborsClassifier</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M992.714,-156.407C1096.89,-144.868 1370.9,-114.52 1505.87,-99.5707\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1506.57,-103.014 1516.13,-98.4343 1505.8,-96.0565 1506.57,-103.014\"/>\n",
       "</g>\n",
       "<!-- max_iter -->\n",
       "<g id=\"node3\" class=\"node\"><title>max_iter</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"42.2463\" cy=\"-18\" rx=\"42.4939\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"42.2463\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">max_iter</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;max_iter -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>LinearSVC&#45;&gt;max_iter</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M289.05,-81.2875C242.346,-71.8302 162.103,-54.7328 94.2463,-36 90.8473,-35.0617 87.3405,-34.038 83.8288,-32.9734\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"84.5927,-29.5452 74.0039,-29.9019 82.504,-36.2264 84.5927,-29.5452\"/>\n",
       "</g>\n",
       "<!-- penalty -->\n",
       "<g id=\"node4\" class=\"node\"><title>penalty</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"139.246\" cy=\"-18\" rx=\"36.2938\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"139.246\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">penalty</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;penalty -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>LinearSVC&#45;&gt;penalty</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M289.098,-74.6935C260.119,-64.0787 219.69,-49.2174 184.246,-36 181.85,-35.1064 179.383,-34.1832 176.896,-33.2505\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"177.91,-29.8924 167.318,-29.6471 175.445,-36.4441 177.91,-29.8924\"/>\n",
       "</g>\n",
       "<!-- dual -->\n",
       "<g id=\"node5\" class=\"node\"><title>dual</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"220.246\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"220.246\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">dual</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;dual -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>LinearSVC&#45;&gt;dual</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M301.826,-71.8761C285.387,-61.2209 264.286,-47.5443 247.663,-36.7701\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"249.242,-33.6226 238.947,-31.1206 245.435,-39.4967 249.242,-33.6226\"/>\n",
       "</g>\n",
       "<!-- loss -->\n",
       "<g id=\"node6\" class=\"node\"><title>loss</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"292.246\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"292.246\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">loss</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;loss -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>LinearSVC&#45;&gt;loss</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M319.347,-71.6966C315.082,-63.4033 309.892,-53.3109 305.199,-44.1866\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"308.235,-42.4367 300.549,-35.1446 302.01,-45.6382 308.235,-42.4367\"/>\n",
       "</g>\n",
       "<!-- C -->\n",
       "<g id=\"node7\" class=\"node\"><title>C</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"364.246\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"364.246\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">C</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;C -->\n",
       "<g id=\"edge5\" class=\"edge\"><title>LinearSVC&#45;&gt;C</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M337.145,-71.6966C341.41,-63.4033 346.601,-53.3109 351.293,-44.1866\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"354.482,-45.6382 355.943,-35.1446 348.257,-42.4367 354.482,-45.6382\"/>\n",
       "</g>\n",
       "<!-- multi_class -->\n",
       "<g id=\"node8\" class=\"node\"><title>multi_class</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"460.246\" cy=\"-18\" rx=\"51.1914\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"460.246\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">multi_class</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;multi_class -->\n",
       "<g id=\"edge6\" class=\"edge\"><title>LinearSVC&#45;&gt;multi_class</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M360.537,-71.8761C379.777,-61.6731 404.241,-48.6997 424.122,-38.157\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"425.988,-41.1289 433.183,-33.3517 422.709,-34.9446 425.988,-41.1289\"/>\n",
       "</g>\n",
       "<!-- random_state -->\n",
       "<g id=\"node9\" class=\"node\"><title>random_state</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"588.246\" cy=\"-18\" rx=\"58.4896\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"588.246\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">random_state</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;random_state -->\n",
       "<g id=\"edge7\" class=\"edge\"><title>LinearSVC&#45;&gt;random_state</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M367.452,-78.4445C412.311,-66.3671 485.979,-46.5335 535.76,-33.131\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"536.945,-36.4366 545.691,-30.4572 535.125,-29.6773 536.945,-36.4366\"/>\n",
       "</g>\n",
       "<!-- RandomForestClassifier&#45;&gt;random_state -->\n",
       "<g id=\"edge15\" class=\"edge\"><title>RandomForestClassifier&#45;&gt;random_state</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M876.447,-77.1228C816.428,-67.2284 730.13,-52.1828 655.246,-36 650.58,-34.9916 645.746,-33.8861 640.909,-32.738\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"641.573,-29.2976 631.03,-30.3391 639.922,-36.1 641.573,-29.2976\"/>\n",
       "</g>\n",
       "<!-- n_estimators -->\n",
       "<g id=\"node11\" class=\"node\"><title>n_estimators</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"720.246\" cy=\"-18\" rx=\"55.7903\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"720.246\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">n_estimators</text>\n",
       "</g>\n",
       "<!-- RandomForestClassifier&#45;&gt;n_estimators -->\n",
       "<g id=\"edge9\" class=\"edge\"><title>RandomForestClassifier&#45;&gt;n_estimators</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M895.033,-71.9656C856.702,-60.3503 806.42,-45.1133 769.631,-33.9651\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"770.219,-30.4861 759.634,-30.9356 768.189,-37.1853 770.219,-30.4861\"/>\n",
       "</g>\n",
       "<!-- criterion -->\n",
       "<g id=\"node12\" class=\"node\"><title>criterion</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"834.246\" cy=\"-18\" rx=\"40.0939\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"834.246\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">criterion</text>\n",
       "</g>\n",
       "<!-- RandomForestClassifier&#45;&gt;criterion -->\n",
       "<g id=\"edge10\" class=\"edge\"><title>RandomForestClassifier&#45;&gt;criterion</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M922.625,-71.8761C905.572,-61.6737 883.89,-48.7013 866.269,-38.159\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"867.849,-35.0258 857.471,-32.8951 864.255,-41.0328 867.849,-35.0258\"/>\n",
       "</g>\n",
       "<!-- max_features -->\n",
       "<g id=\"node13\" class=\"node\"><title>max_features</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"951.246\" cy=\"-18\" rx=\"58.4896\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"951.246\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">max_features</text>\n",
       "</g>\n",
       "<!-- RandomForestClassifier&#45;&gt;max_features -->\n",
       "<g id=\"edge11\" class=\"edge\"><title>RandomForestClassifier&#45;&gt;max_features</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M951.246,-71.6966C951.246,-63.9827 951.246,-54.7125 951.246,-46.1124\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"954.746,-46.1043 951.246,-36.1043 947.746,-46.1044 954.746,-46.1043\"/>\n",
       "</g>\n",
       "<!-- min_samples_split -->\n",
       "<g id=\"node14\" class=\"node\"><title>min_samples_split</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"1105.25\" cy=\"-18\" rx=\"77.1866\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1105.25\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">min_samples_split</text>\n",
       "</g>\n",
       "<!-- RandomForestClassifier&#45;&gt;min_samples_split -->\n",
       "<g id=\"edge12\" class=\"edge\"><title>RandomForestClassifier&#45;&gt;min_samples_split</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M988.919,-71.8761C1011.15,-61.773 1039.35,-48.9536 1062.42,-38.4675\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1063.98,-41.6037 1071.63,-34.2793 1061.08,-35.2312 1063.98,-41.6037\"/>\n",
       "</g>\n",
       "<!-- min_samples_leaf -->\n",
       "<g id=\"node15\" class=\"node\"><title>min_samples_leaf</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"1276.25\" cy=\"-18\" rx=\"75.2868\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1276.25\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">min_samples_leaf</text>\n",
       "</g>\n",
       "<!-- RandomForestClassifier&#45;&gt;min_samples_leaf -->\n",
       "<g id=\"edge13\" class=\"edge\"><title>RandomForestClassifier&#45;&gt;min_samples_leaf</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1025.81,-72.9406C1082.15,-60.8061 1158.67,-44.3251 1212.24,-32.7852\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1213.27,-36.1452 1222.31,-30.618 1211.79,-29.3021 1213.27,-36.1452\"/>\n",
       "</g>\n",
       "<!-- bootstrap -->\n",
       "<g id=\"node16\" class=\"node\"><title>bootstrap</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"1413.25\" cy=\"-18\" rx=\"43.5923\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1413.25\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">bootstrap</text>\n",
       "</g>\n",
       "<!-- RandomForestClassifier&#45;&gt;bootstrap -->\n",
       "<g id=\"edge14\" class=\"edge\"><title>RandomForestClassifier&#45;&gt;bootstrap</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1025.84,-83.4271C1108.37,-76.316 1244.94,-61.7389 1360.25,-36 1363.9,-35.1836 1367.67,-34.2243 1371.43,-33.1843\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1372.67,-36.4676 1381.27,-30.2854 1370.69,-29.7526 1372.67,-36.4676\"/>\n",
       "</g>\n",
       "<!-- n_neighbors -->\n",
       "<g id=\"node18\" class=\"node\"><title>n_neighbors</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"1528.25\" cy=\"-18\" rx=\"53.8905\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1528.25\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">n_neighbors</text>\n",
       "</g>\n",
       "<!-- KNeighborsClassifier&#45;&gt;n_neighbors -->\n",
       "<g id=\"edge17\" class=\"edge\"><title>KNeighborsClassifier&#45;&gt;n_neighbors</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1569.65,-71.6966C1563.03,-63.2653 1554.94,-52.9748 1547.68,-43.7323\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1550.24,-41.3231 1541.31,-35.6223 1544.73,-45.6479 1550.24,-41.3231\"/>\n",
       "</g>\n",
       "<!-- weights -->\n",
       "<g id=\"node19\" class=\"node\"><title>weights</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"1638.25\" cy=\"-18\" rx=\"38.1938\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1638.25\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">weights</text>\n",
       "</g>\n",
       "<!-- KNeighborsClassifier&#45;&gt;weights -->\n",
       "<g id=\"edge18\" class=\"edge\"><title>KNeighborsClassifier&#45;&gt;weights</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1596.84,-71.6966C1603.63,-63.0541 1611.96,-52.4579 1619.36,-43.0394\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1622.14,-45.1702 1625.56,-35.1446 1616.63,-40.8454 1622.14,-45.1702\"/>\n",
       "</g>\n",
       "<!-- p -->\n",
       "<g id=\"node20\" class=\"node\"><title>p</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"1721.25\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1721.25\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">p</text>\n",
       "</g>\n",
       "<!-- KNeighborsClassifier&#45;&gt;p -->\n",
       "<g id=\"edge19\" class=\"edge\"><title>KNeighborsClassifier&#45;&gt;p</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1617,-71.8761C1639.66,-60.382 1669.26,-45.372 1691.12,-34.2821\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1692.93,-37.2856 1700.27,-29.6404 1689.77,-31.0428 1692.93,-37.2856\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f0ff00fddd8>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CS = hdl2cs(HDL)\n",
    "g = plot_hdl(HDL)\n",
    "g.graph_attr['size'] = \"15,8\"\n",
    "g"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "### 定义一个简单的AutoML评价器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "在定义好了配置空间之后，现在我们来看评价器。\n",
    "\n",
    "在UltraOpt的设计哲学中，评价器可以是一个类（实现`__call__`魔法方法），也可以是个函数。但其必须满足：\n",
    "\n",
    "- 接受 `dict` 类型参数 `config` \n",
    "- 返回 `float` 类型参数 `loss` \n",
    "\n",
    "在AutoML问题中，评价器的工作流程如下：\n",
    "1. 将config转化为一个机器学习模型\n",
    "2. 在训练集上对机器学习模型进行训练\n",
    "3. 在验证集上得到相应的评价指标\n",
    "4. 对评价指标进行处理，使其`越小越好`，返回`loss`\n",
    "\n",
    "在了解了这些知识后，我们来开发 **AutoML 评价器**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "为了方便同学们理解，我们先顺序地将之前提到的**AutoML 评价器**工作流程跑一遍：\n",
    "\n",
    "#### Step 1. 将config转化为一个机器学习模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "# 引入 sklearn 的分类器\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "假设评价器传入了一个 `config` 参数，我们先通过获取配置空间默认值（也可以对配置空间 `CS` 对象进行采样）得到 `config`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'classifier:__choice__': 'LinearSVC',\n",
       " 'classifier:LinearSVC:C': 1.0,\n",
       " 'classifier:LinearSVC:dual': 'True:bool',\n",
       " 'classifier:LinearSVC:loss': 'squared_hinge',\n",
       " 'classifier:LinearSVC:max_iter': 600,\n",
       " 'classifier:LinearSVC:multi_class': 'ovr',\n",
       " 'classifier:LinearSVC:penalty': 'l2',\n",
       " 'classifier:LinearSVC:random_state': '42:int'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config = CS.get_default_configuration().get_dictionary()\n",
    "config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "我们用**配置分层函数** `ultraopt.hdl.layering_config` 处理这个配置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'classifier': {'LinearSVC': {'C': 1.0,\n",
       "   'dual': True,\n",
       "   'loss': 'squared_hinge',\n",
       "   'max_iter': 600,\n",
       "   'multi_class': 'ovr',\n",
       "   'penalty': 'l2',\n",
       "   'random_state': 42}}}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layered_dict = layering_config(config)\n",
    "layered_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n",
       " -->\n",
       "<!-- Title: Layered Dict Pages: 1 -->\n",
       "<svg width=\"655pt\" height=\"188pt\"\n",
       " viewBox=\"0.00 0.00 655.49 188.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 184)\">\n",
       "<title>Layered Dict</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-184 651.495,-184 651.495,4 -4,4\"/>\n",
       "<!-- classifier -->\n",
       "<g id=\"node1\" class=\"node\"><title>classifier</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"290.5,-180 225.5,-180 225.5,-144 290.5,-144 290.5,-180\"/>\n",
       "<text text-anchor=\"middle\" x=\"258\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\">classifier</text>\n",
       "</g>\n",
       "<!-- LinearSVC -->\n",
       "<g id=\"node2\" class=\"node\"><title>LinearSVC</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"297,-108 219,-108 219,-72 297,-72 297,-108\"/>\n",
       "<text text-anchor=\"middle\" x=\"258\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">LinearSVC</text>\n",
       "</g>\n",
       "<!-- classifier&#45;&gt;LinearSVC -->\n",
       "<g id=\"edge8\" class=\"edge\"><title>classifier&#45;&gt;LinearSVC</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M258,-143.697C258,-135.983 258,-126.712 258,-118.112\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"261.5,-118.104 258,-108.104 254.5,-118.104 261.5,-118.104\"/>\n",
       "</g>\n",
       "<!-- C -->\n",
       "<g id=\"node3\" class=\"node\"><title>C</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">C</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;C -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>LinearSVC&#45;&gt;C</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M218.892,-80.5091C179.065,-71.3674 115.83,-55.4811 63,-36 61.1518,-35.3185 59.2689,-34.578 57.382,-33.8001\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"58.6185,-30.5205 48.0555,-29.6992 55.8008,-36.9284 58.6185,-30.5205\"/>\n",
       "</g>\n",
       "<!-- dual -->\n",
       "<g id=\"node4\" class=\"node\"><title>dual</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"99\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"99\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">dual</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;dual -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>LinearSVC&#45;&gt;dual</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M218.796,-72.5127C194.562,-62.2205 162.869,-48.58 135,-36 133.205,-35.1895 131.363,-34.3489 129.507,-33.4947\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"130.807,-30.2392 120.264,-29.1893 127.851,-36.5845 130.807,-30.2392\"/>\n",
       "</g>\n",
       "<!-- loss -->\n",
       "<g id=\"node5\" class=\"node\"><title>loss</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"171\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"171\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">loss</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;loss -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>LinearSVC&#45;&gt;loss</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M236.494,-71.6966C224.111,-61.7332 208.501,-49.1733 195.627,-38.8148\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"197.609,-35.917 187.623,-32.3751 193.22,-41.3708 197.609,-35.917\"/>\n",
       "</g>\n",
       "<!-- max_iter -->\n",
       "<g id=\"node6\" class=\"node\"><title>max_iter</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"258\" cy=\"-18\" rx=\"42.4939\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"258\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">max_iter</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;max_iter -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>LinearSVC&#45;&gt;max_iter</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M258,-71.6966C258,-63.9827 258,-54.7125 258,-46.1124\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"261.5,-46.1043 258,-36.1043 254.5,-46.1044 261.5,-46.1043\"/>\n",
       "</g>\n",
       "<!-- multi_class -->\n",
       "<g id=\"node7\" class=\"node\"><title>multi_class</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"370\" cy=\"-18\" rx=\"51.1914\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"370\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">multi_class</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;multi_class -->\n",
       "<g id=\"edge5\" class=\"edge\"><title>LinearSVC&#45;&gt;multi_class</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M285.398,-71.8761C301.026,-62.1089 320.716,-49.8028 337.167,-39.5207\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"339.302,-42.3137 345.927,-34.0456 335.592,-36.3777 339.302,-42.3137\"/>\n",
       "</g>\n",
       "<!-- penalty -->\n",
       "<g id=\"node8\" class=\"node\"><title>penalty</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"476\" cy=\"-18\" rx=\"36.2938\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"476\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">penalty</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;penalty -->\n",
       "<g id=\"edge6\" class=\"edge\"><title>LinearSVC&#45;&gt;penalty</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M297.134,-77.6104C332.125,-67.268 384.718,-51.3311 430,-36 432.722,-35.0785 435.525,-34.1045 438.342,-33.1079\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"439.646,-36.3582 447.868,-29.6756 437.274,-29.7726 439.646,-36.3582\"/>\n",
       "</g>\n",
       "<!-- random_state -->\n",
       "<g id=\"node9\" class=\"node\"><title>random_state</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"589\" cy=\"-18\" rx=\"58.4896\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"589\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">random_state</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;random_state -->\n",
       "<g id=\"edge7\" class=\"edge\"><title>LinearSVC&#45;&gt;random_state</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M297.331,-81.6608C348.854,-71.9122 441.942,-53.8381 521,-36 525.722,-34.9345 530.619,-33.7896 535.522,-32.6158\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"536.649,-35.9438 545.54,-30.1818 534.997,-29.1417 536.649,-35.9438\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f0ff4c7fe10>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plot_layered_dict(layered_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "我们需要获取这个配置的如下信息：\n",
    "\n",
    "- 算法选择的结果\n",
    "- 被选择算法对应的参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'LinearSVC'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "AS_HP = layered_dict['classifier'].copy()\n",
    "AS, HP = AS_HP.popitem()\n",
    "AS  # 算法选择的结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'C': 1.0,\n",
       " 'dual': True,\n",
       " 'loss': 'squared_hinge',\n",
       " 'max_iter': 600,\n",
       " 'multi_class': 'ovr',\n",
       " 'penalty': 'l2',\n",
       " 'random_state': 42}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HP  # 被选择算法对应的参数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "根据 **算法选择结果** + **对应的参数** 实例化一个 **机器学习对象**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearSVC(C=1.0, class_weight=None, dual=True, fit_intercept=True,\n",
       "          intercept_scaling=1, loss='squared_hinge', max_iter=600,\n",
       "          multi_class='ovr', penalty='l2', random_state=42, tol=0.0001,\n",
       "          verbose=0)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ML_model = eval(AS)(**HP)\n",
    "ML_model # 实例化的机器学习对象"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "#### Step 2. 在训练集上对机器学习模型进行训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "我们采用MNIST手写数字数据集的一个子集来作为训练数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits\n",
    "import seaborn as sns\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "X, y = load_digits(return_X_y=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAAD8CAYAAABAWd66AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU1f3/8ddnskAChLUICVSwKMWvKCgiiKWgFRQBsSCLdcNaEKxF/SlSRG3dV1SqaBEQLIhQF1BAwFIVQamggECUJYCShEXWsJNMPr8/MokBswwhc+/J8Hn6uA9m7iznbebOJyfnnnuvqCrGGGP8F/A7gDHGmDxWkI0xxhFWkI0xxhFWkI0xxhFWkI0xxhFWkI0xxhFWkI0xphgiMl5EtovIqkLr/iYiGSKyPLR0Kea1V4jIGhFZLyLDwmrP5iEbY0zRRKQ9sB94Q1XPCa37G7BfVZ8t4XUxwFrgciAdWAL0U9XUktqzHrIxxhRDVRcAu8rw0tbAelXdoKpHgbeAq0t7UWwZGjohsfEp1gUP+Xv9Dn5H4O4xbf2OAEDWE2/7HYHzV/zodwQAth3Y43cEZ+QczZCTfY/sHRvCrjnxv/jVQGBAoVVjVHVMGC/9s4jcCCwF/p+q7j7u8RRgc6H76cBFpb2p9ZCNMacsVR2jqq0KLeEU41eAXwEtgC3Ac+WVJ+I9ZGOM8VRuMKJvr6rb8m+LyGvAzCKelgE0LHS/QWhdiawgG2OiSzAnom8vIvVVdUvo7jXAqiKetgQ4U0Qak1eI+wLXlfbeVpCNMVFFNbfc3ktEpgAdgDoikg48BHQQkRaAApuAgaHnJgNjVbWLquaIyJ+BuUAMMF5VV5fWnhVkY0x0yS2/gqyq/YpYPa6Y52YCXQrdnw3MPpH2rCAbY6JLOfaQvWYF2RgTXSK8Uy+SrCAbY6KL9ZCNMcYNGuFZFpHk9IEhnTt1YPWqBXyXupCh995+yue4feEL/Gnuk9w6+3Fu+eART9p8aPJ/6Dh8LD2fmFyw7uVZi7n2yTfp/dQUbnt5Otv37vckS76Enj2p/frr1H79dRJ79fK07XzP/eMRVqxdwPzPp/vSfj5Xtk1XcgB5O/XCXRzjbEEOBAKMevExuna7nubndaRPnx40a3bmKZsj36S+jzK2y3DGd3vAk/a6X9SM0YO6H7PupkvP59/DrmPaff1of05jxsxZ4kkWgJjGjUns2pWdt93GzltvJb5tW2JSUjxrP9+0KdP5Q6+BnrdbmCvbpis5Cmhu+ItjSi3IIvJrEblPREaFlvtEpFmkg7W+sCVpaZvYuPEHsrOzmTZtBt27dY50s87m8MsFTVJISqx8zLqqCfEFtw8dyeakTz5wAmJ/+UuyU1PhyBEIBslevpxKv/mNhwny/O/zr9ize6/n7RbmyrbpSo4CucHwF8eUWJBF5D7yzlIkwJehRYAp4Z7fs6ySU+qxOT2z4H56xhaSk+tFskmnc+RRrps0jFtmPkrLfh19ypDnHzO/oPODrzP7qzUM6tLGs3ZzNm4k7txzkaQkqFSJ+DZtiKlb17P2XeLKtulKjgIVuIdc2k69PwL/p6rZhVeKyEhgNfBkUS8SkQGEzqAkMdUJBKqUQ1TzRs+H2bdtN4m1k7hu0jB2pG1h85ff+ZLljq5tuaNrW8bNW8pbn61gsEdFOfjDDxyYMoWazzyDHj5Mzvr1qINjgcZHUbxTLxdILmJ9/dBjRSp8BqWyFuPMjK00bPBT0w1S6pOZubVM73UyXMkBsG9b3hn+Du7MYs3cpSS3OMOXHIV1adWU+SvSPG3z8OzZ7Bo4kN1DhqD79hHcvLn0F0UhV7ZNV3IUiOKdencC80XkQxEZE1rmAPOBIZEMtmTpcpo0aUyjRg2Ji4ujd++r+WDmvEg26XSOuIRKxFepXHD7jPbN+XFNuuc5AL7f/tP5ez9ZuYHGdWt62r7UqAFAoG5dKrVvz+H58z1t3xWubJuu5MinGgx7cU2JQxaqOkdEziLv7Pf5u7IzgCUa4f+bYDDIkDtHMHvWm8QEAkyYOJXU1LWRbNLpHFXqJNFrzF0ABGJjWD3jczZ8+k3E2x02YQ5L12ewZ/9hOj0wnkFdLmJh6vds2r6bgAj1a1bj/j7ejmfXePhhAklJaE4O+154Ad3v7bQ7gJfHPkPbdhdSq3YNlq6az7NPvsxbk971NIMr26YrOQo4ODYcrohfU8+uGPITu2LIT+yKIT+xK4b8pDyuGHL46/fDrjmVz+/u5SShUtmResaY6FKBe8hWkI0x0SWYXfpzHGUF2RgTXRycPREuK8jGmOhiQxbGGOMI6yEbY4wjrCAbY4wb1HbqGWOMI2wM2YTDhYMyYi/oUvqTPFC5sb8ndgfY9PYHfkcA4OkLvDm3dUke2vKJ3xHKjw1ZGGOMI6yHbIwxjrAesjHGOMJ6yMYY44icinuCeivIxpjoYj1kY4xxRDmOIYvIeKArsF1VzwmtewboBhwF0oD+qvqzc6iKyCZgHxAEclS1VWntlXrVaWOMqVDK9yKnE4Arjlv3EXCOqp4LrAX+WsLrO6pqi3CKMVhBNsZEm3K8pp6qLgB2HbdunqrmD1QvBhqUV3QryMaY6FK+PeTS3AJ8WFwSYJ6IfCUiA8J5M6fHkDt36sDIkQ8TEwgw/vUpPP3My6dUjocm/4cFqzdRq1oC7/z1DwC8PGsxn6zcgIhQq2oCD1//O+pWrxqxDCMeH8mCRV9Sq2YNpk96NS/DuEm88/4cataoDsCQgTfR/uLWEcsAkHDLPcSedxGatYf9D/wJAKlSjYRBIwjUOY3cHds4OPoROBi56+u58rMo7PaFL3D0wGE0mEtuMMj4bv4c9efKdxU4oVkWoUJZuFiOUdUxYb72fiAHmFzMUy5R1QwRqQt8JCLfhXrcxXK2IAcCAUa9+BhXdOlHevoWFn8xmw9mzuPbb9edMjm6X9SMvu3PZcSkjwrW3XTp+dx+VRsA3vx0BWPmLGFEBC8y2qPL5VzXszvDH3n2mPU39OlB/+t6Razd4x1dOJcj86eTeOt9BesqdelLMHUZB2e/RaUufal8VV8O/3tsxDK48rM43qS+j3Jot/cXes3nyne1wAlcJzRUfMMqwIWJyM3k7ey7TIu5MKmqZoT+3S4i75F3segSC7KzQxatL2xJWtomNm78gezsbKZNm0H3bp1PqRwXNEkhKbHyMeuqJsQX3D50JJtIX6GxVYvmVE+qFuFWShdcuxLdv++YdbEtL+boorzLzR9dNI/Ylu0imsGVn4VrXPmuFijHMeSiiMgVwFCgu6oeLOY5VUSkWv5toBOwqrT3LnNBFpH+ZX1tOJJT6rE5PbPgfnrGFpKT60WySadzFPaPmV/Q+cHXmf3VGgZ1aeNLhinvfMA1Nw5ixOMj2Zu1r/QXRECgek10b97+Ft27i0D1mr7k8PdnoVw3aRi3zHyUlv0i95dSSZz7jpRjQRaRKcAXQFMRSReRPwIvAdXIG4ZYLiKvhp6bLCKzQy89DVgoIiuAL4FZqjqntPZOpof89xL+JwaIyFIRWZqbe+AkmjBFuaNrW+Y+3J8uFzTlrc9WeN5+n2uu4sNp43lnwsv8onYtnnnpNc8zFOkE/lQtL37/LN7o+TDjrhrBWzc9zQU3Xk7D1r/2tH0nleNOPVXtp6r1VTVOVRuo6jhVbaKqDUPT2Vqo6m2h52aqapfQ7Q2qel5o+T9VfSyc6CUWZBH5pphlJXm/AYr7nxijqq1UtVUgUCWcHD+TmbGVhg2SC+43SKlPZubWMr3XyXAlR1G6tGrK/BVpnrdbp1ZNYmJiCAQC9Op+JatS13qeASB3726kei0ApHotcrN+Njc/4vz+WezbthuAgzuzWDN3KcktzvC0fXDwOxIMhr84prQe8mnAjeQdlXL8sjOSwZYsXU6TJo1p1KghcXFx9O59NR/MnBfJJp3Oke/77T8VnU9WbqBxXe//TP9xx0/TMud/+jlNzjjd8wwAOcu/IL5dJwDi23UiZ9nnnmfw82cRl1CJ+CqVC26f0b45P65J96z9fK59RyI9hhxJpc2ymAlUVdXlxz8gIp9EJFFIMBhkyJ0jmD3rTWICASZMnEqqDz0xP3MMmzCHpesz2LP/MJ0eGM+gLhexMPV7Nm3fTUCE+jWrcX8EZ1gA3PvQkyxZ9g179mRxWY/rGfzHG1iy7BvWrNsAAin1TuOhoX+JaAaAhIHDif31eUjV6lR7bgqHp0/kyKy3SBw8grj2V6A7tnPwlUcimsGVn0W+KnWS6DXmLgACsTGsnvE5Gz79xrP287nyXS3gYKENlxQzY6PcxManeD+w56h9H5R0hKU3XLliyMF7w5onH1GJz5zwbKeIsCuG/CTnaMZJTxw6NPbusGtOwq0jIz1R6YQ4Ow/ZGGPKQnMrbh/QCrIxJrpU4CELK8jGmOji4OyJcFlBNsZEF+shG2OMI6wgG2OMI3w4YrO8WEE2xkQX6yEbY4wjbNqb266s19LvCIAbB2W0bX6T3xEAWL5zg98R2Hv6/X5HAOD38Xv9jsBDfgcoTzbLwhhj3KA2ZGGMMY6wIQtjjHFE+Vy81BdWkI0x0cV6yMYY44gc26lnjDFusCELY4xxhA1ZGGOMG2zamzHGuKIC95BLu8iprzp36sDqVQv4LnUhQ++93ZcMcZXieO79kYya8w9e/s/LXHf3dZ61PeLxkbS/qi89rr+tYN3L4yZx6dXX0/Om2+l50+0s+PxLz/KcllyXV99+kWmf/oupn7xB31t7edZ2YX5tF/FXDyTx3ldJGPx0wbqYsy8iYfAzJD40mUCy91d8rnVzD86Y/QqNZ40m+fmhSHyc5xnAje9qgVwNf3GMsz3kQCDAqBcf44ou/UhP38LiL2bzwcx5fPvtOk9zZB/J5v6+wzl88DAxsTE89c7TfPXxV6xZtibibffocjnX9ezO8EeePWb9DX160P8674thTk6Q5//+MmtWriWxSgL/mjuO/y1Yysa1mzzL4Od2kbP8U3K+nEulawYXrMvdvpnDU0dSqdutEW//eLGn1abmjd3ZcOVt6JGjpLz4V5K6/pa97/7H0xyufFcLVOBDp53tIbe+sCVpaZvYuPEHsrOzmTZtBt27dfYly+GDhwGIjY0lNjaGSF8YNl+rFs2pnlTNk7bCsXP7TtaszLua8MEDh9i0bhN169XxNIOf20Xu99+hh/Yfs053ZKI7t3jSflEkNgapHA8xASShEjnbd3qewaXvKuRdUy/cxTWlFmQR+bWIXCYiVY9bf0XkYkFySj02p2cW3E/P2EJycr1INlmsQCDAix+O4l/LJrFs4XLWLvfxEufAlHc+4JobBzHi8ZHszdrnS4b6DerRtPlZrPo61dN2Xdou/JazbSc7x73LmZ9O5MzPJ5O77wAHFi7zPIdzn0kFHrIosSCLyF+AGcAdwCoRubrQw4+X8LoBIrJURJbm5h4on6Q+ys3NZciVf6H/RTdz1nln8cuzTvctS59rruLDaeN5Z8LL/KJ2LZ556TXPMyQkJvD0uEd57sFRHNh/0PP2TZ5AUlWqXdaG9Zf2Z1276wkkVCape0e/Y/kvNzf8xTGl9ZD/BFygqj2ADsADIjIk9JgU9yJVHaOqrVS1VSBQpUzBMjO20rBBcsH9Bin1yczcWqb3Ki8Hsg6w8otvuKDD+b5lqFOrJjExMQQCAXp1v5JVqd721mNiY3h63KPMefcjPp69wNO2wc3twi9VLm5BdvpWgruyICfIvnmLSDy/mec5nPtMorWHDARUdT+Aqm4iryhfKSIjKaEgl4clS5fTpEljGjVqSFxcHL17X80HM+dFsskiJdVKokpS3i+V+ErxtPhNS9LT0j3Pke/HHbsKbs//9HOanOFtb/3BkcPYuG4Tk/851dN287myXbgge8uPJLT4NVK5EgCJbVtwJG2z5zmc+0zKsSCLyHgR2S4iqwqtqyUiH4nIutC/NYt57U2h56wTkbBORF7aLIttItJCVZcDqOp+EekKjAeah9NAWQWDQYbcOYLZs94kJhBgwsSppHrcGwSoVbcWd468i0BMgEAgwMKZn7Fk/hJP2r73oSdZsuwb9uzJ4rIe1zP4jzewZNk3rFm3AQRS6p3GQ0P/4kkWgPNaN+eqa69gXWoakz8aD8DoJ8aw6L+LPcvg53ZRqecdBBo1QxKrkXD3S2R//DZ6aD/xXW5GEpOofN1Qgls3cWTSk57kObxiDVlzFtJ4+ig0GORI6gb2TP3Qk7YLc+W7mk+D5ToUMQF4CXij0LphwHxVfVJEhoXu31f4RSJSi7zz/rcCFPhKRN5X1d0lNSYlzRgQkQZAjqr+7O8PEWmnqotK+7+JjU/x/e8CV64Y8u7Xo/yOYFcMKWTv8PZ+RwDgh8n+XzGk+fcr/I4AQM7RjJP+yzvrj5eHXXOSxn1Uansi0giYqarnhO6vATqo6hYRqQ98oqpNj3tNv9BzBobu/zP0vCkltVViD1lVi/3bPJxibIwxXjuR6WwiMgAYUGjVGFUdU8rLTlPV/LmOW4HTinhOClB4/Cg9tK5Ezh4YYowxZXICBTlUfEsrwCW9XkWk3EYBnD0wxBhjyiT3BJay2RYaqiD07/YinpMBNCx0v0FoXYmsIBtjoorm5Ia9lNH7QP7OmJvIO1bjeHOBTiJSMzQLo1NoXYmsIBtjoks59pBFZArwBdBURNJF5I/Ak8DlIrIO+F3oPiLSSkTGAqjqLuARYEloeTi0rkQ2hmyMiSrleY4KVe1XzEOXFfHcpcCthe6PJ2+KcNisIBtjoot7R0SHzQqyMSaquHgWt3CdEgW5QSDR7wgA5MwY7XcEJw7IcIULB2SYCLAesjHGuEFz/E5QdlaQjTFRRa2HbIwxjrCCbIwxbrAesjHGOMIKsjHGOEKDEb12RkRZQTbGRBXrIRtjjCM013rIxhjjhIrcQ3b6bG+dO3Vg9aoFfJe6kKH33u5rFgkIw2c9xeBx95X+5HLy0IfL6fjSXHqO/6Rg3ciPU+kx9r9c+/on3PXeErIOZ3uWB9z4TFzIAFDr5h6cMfsVGs8aTfLzQ5H4uFMyA7jzmQCoStiLa5wtyIFAgFEvPkbXbtfT/LyO9OnTg2bNzvQtz6X9u7B1fannly5X3c9pyOheFx2zrk2jOrx9Swf+3b8Dp9eswvjF6zzL48Jn4kIGgNjTalPzxu5svGYIG68ajARiSOr621MuA7jzmeTT3PAX1zhbkFtf2JK0tE1s3PgD2dnZTJs2g+7dOvuSpUa9Wpxz6fksemu+p+1e0LA2SQnxx6y7uHFdYgN5H9u5yTXZtu+wZ3lc+ExcyJBPYmOQyvEQE0ASKpGzfecpmcGlzwQgNyhhL64ptSCLSGsRuTB0+2wRuVtEukQ6WHJKPTanZxbcT8/YQnJyvUg3W6RrH7yZ956YRG4JV+j2w/SVm7nkjLqetefCZ+JCBoCcbTvZOe5dzvx0Imd+PpncfQc4sHDZKZcB3PlM8mmuhL24psSCLCIPAaOAV0TkCeAloAowTETuL+F1A0RkqYgszc09UK6BvXbOpeezb+defli10e8ox3jti7XEBIQuZ5d6IVsTAYGkqlS7rA3rL+3PunbXE0ioTFL3jqdcBhdV5IJc2iyLXkALoBJ5l7tuoKpZIvIs8D/gsaJeVPhKrrHxKWXqVmZmbKVhg+SC+w1S6pOZubUsb3VSftWqKef+rhXndGxJbKV4EqomcPPzdzDhrn94niXfjJWb+SxtO//s0wYR7zYqFz4TFzIAVLm4BdnpWwnuygJg37xFJJ7fjKz3Pz6lMoA7n0k+x/6QPSGlDVnkqGpQVQ8CaaqaBaCqh4jwKTyWLF1OkyaNadSoIXFxcfTufTUfzJwXySaLNOPpKQxvO4gRl/yZcXe8wJrPV/lajBdt2M7EL9fzwu8vJCHO21mLLnwmLmQAyN7yIwktfo1UrgRAYtsWHEnbfMplAHc+k3zR3EM+KiKJoYJ8Qf5KEalOhAtyMBhkyJ0jmD3rTWICASZMnEpq6tpINumcYe9/xdLNO9lz6CidRn/EoEuaMn7xOo4Gc7lt2mIAzq1fkxGdz/UkjwufiQsZAA6vWEPWnIU0nj4KDQY5krqBPVM/POUygDufST4Xp7OFS7SE/r2IVFLVI0WsrwPUV9WVpTVQ1iGL8vSn5HZ+RwBg5AO/9DsC1QZN8TuCM1aefp7fEZzR/PsVfkcAIOdoxklX07XNrgi75pz17RynqneJPeSiinFo/Q5gR0QSGWPMSajIPWQ7dNoYE1VcHBsOlxVkY0xUqcizLKwgG2OiivWQjTHGEcFcZ88IUSoryMaYqFKRhywq7q8SY4wpQq5K2EtJRKSpiCwvtGSJyJ3HPaeDiOwt9JwHTya79ZCNMVGlvKa9qeoa8k4dgYjEABnAe0U89TNV7VoebVpBNsZElQgNWVxG3ukjvo/Iu4ecEgW5gfpzFYXjbXoy1e8IppBaKQf9jgDAroxEvyNEldKGIgoTkQHAgEKrxoROjna8vkBxh7m2FZEVQCZwj6quDjvAcU6JgmyMOXWcyCyLwmemLI6IxAPdgb8W8fDXwOmquj90nvjpQJkvl2I79YwxUUVPYAnTlcDXqrrtZ22pZqnq/tDt2UBc6Fw/ZWI9ZGNMVDmRIYsw9aOY4QoRqQdsU1UVkdbkdXLLfB0tK8jGmKhSnicXEpEqwOXAwELrbstrR18l7yIeg0QkBzgE9NWSTqFZCivIxpioUp4nalfVA0Dt49a9Wuj2S+Rd2q5cWEE2xkQVxc5lYYwxTsix8yEbY4wbrIdsjDGOiOjFPiPM6XnInTt1YPWqBXyXupCh997uW47bF77An+Y+ya2zH+eWDx7xJUOtm3twxuxXaDxrNMnPD0Xi/Tn60IXPxIUMAAk9e1L79dep/frrJPbq5UsG2y5+TpGwF9c4W5ADgQCjXnyMrt2up/l5HenTpwfNmpX5AJiTNqnvo4ztMpzx3R7wvO3Y02pT88bubLxmCBuvGowEYkjq+lvPc7jwmbiQASCmcWMSu3Zl5223sfPWW4lv25aYlBRPM9h2UbTcE1hcc8IFWUTeiESQ47W+sCVpaZvYuPEHsrOzmTZtBt27dfaiaSdJbAxSOR5iAkhCJXK2l3nueZm58Jm4kAEg9pe/JDs1FY4cgWCQ7OXLqfSb33iew7aLnwsiYS+uKXEMWUTeP34V0FFEagCoavdIBUtOqcfm9MyC++kZW2h9YctINVcK5bpJw1CFZZPns2zKx562nrNtJzvHvcuZn04k98hRDiz8mgMLl3maAdz4TFzIAJCzcSNVb70VSUpCjxwhvk0bctas8TaDbRdFqsBXcCp1p14DIBUYS96h3wK0Ap4r6UWFz6AkMdUJBKqcfFIfvdHzYfZt201i7SSumzSMHWlb2Pzld561H0iqSrXL2rD+0v4Esw7QYNRwkrp3JOt9b38xmJ8Ef/iBA1OmUPOZZ9DDh8lZvx7N9faPYNsuipbrYM83XKUNWbQCvgLuB/aq6ifAIVX9VFU/Le5FqjpGVVupaquyFuPMjK00bJBccL9BSn0yM7eW6b1O1r5tuwE4uDOLNXOXktziDE/br3JxC7LTtxLclQU5QfbNW0Ti+c08zQBufCYuZMh3ePZsdg0cyO4hQ9B9+whu3uxp+7ZdFC0CJxfyTIkFWVVzVfV5oD9wv4i8hEdT5ZYsXU6TJo1p1KghcXFx9O59NR/MnOdF08eIS6hEfJXKBbfPaN+cH9eke5ohe8uPJLT4NVK5EgCJbVtwJM3bLz+48Zm4kCGf1KgBQKBuXSq1b8/h+fM9bd+2i6JV5J16YRVXVU0HrhWRq4CsyEbKEwwGGXLnCGbPepOYQIAJE6eSmrrWi6aPUaVOEr3G3AVAIDaG1TM+Z8On33ia4fCKNWTNWUjj6aPQYJAjqRvYM/VDTzOAG5+JCxny1Xj4YQJJSWhODvteeAHdv9/T9m27KFquVNwhCzmJExOFJTY+xfe/DP5ev4PfEQD4ffxuvyPQ/PsVfkdwRsbF/k3NKsyFK4a4sl3kHM046Wo6tf4fwq45fbZMdqp625F6xpioEs2zLIwxpkKpyLMsrCAbY6KK72OkJ8EKsjEmqtiQhTHGOMLF6WzhsoJsjIkqQeshG2OMG6yHbIwxjrCC7Lh0yfY7AgC//EN1vyPA434HyHNalRp+RyDphlZ+RwBg9fAf/I4QVSrwJfVOjYJsjDl1WA/ZGGMcEfQ7wEmwgmyMiSo2D9kYYxxhQxbGGOOIilyQnb3qtDHGlEV5XjFERDaJyEoRWS4iS4t4XERklIisF5FvROT8k8luPWRjTFSJwBhyR1XdUcxjVwJnhpaLgFdC/5aJFWRjTFTxeJbF1cAbmnelj8UiUkNE6qvqlrK8mQ1ZGGOiSi4a9iIiA0RkaaFlwHFvp8A8EfmqiMcAUoDCFzJMD60rE6cLcudOHVi9agHfpS5k6L23+5pFAsLwWU8xeNx9nrUZf/VAEu99lYTBTxesizn7IhIGP0PiQ5MJJHt79Wtw4zN57h+PsGLtAuZ/Pt3Tdh/6cDkdX5pLz/GfFKwb+XEqPcb+l2tf/4S73ltC1mHvjgpN/FV9Lpz/dMHSfv0EGgzo4ln7hbmwXeQ7kYucquoYVW1VaBlz3Ntdoqrnkzc0cbuItI9kdmcLciAQYNSLj9G12/U0P68jffr0oFkz/66Bdmn/Lmxdn+FpmznLP+XwpCePWZe7fTOHp44k9/vvPM0C7nwm06ZM5w+9BnrebvdzGjK617HDg20a1eHtWzrw7/4dOL1mFcYvXudZnoNpW1hy2dC85fL7CB46yo7ZX3rWfj5Xtot85blTT1UzQv9uB94DWh/3lAygYaH7DULryuSECrKIXCIid4tIp7I2GK7WF7YkLW0TGzf+QHZ2NtOmzaB7t86RbrZINerV4pxLz2fRW95e5j33++/QQ8deyVh3ZKI7yzQ8ddJc+Uz+9/lX7Nm91/N2L2hYm6SE+GPWXdy4LrGBvK/Ruck12bbvsOe5AGr9pjmHNm3lcHpx+54ix5XtIt+J9JBLIiJVRKRa/m2gE7DquKe9D9wYmm3RBthb1vFjKKUgi8iXhW7/CXgJqAY8JCLDytpoOJJT6rE5PbPgfm39GDoAABJsSURBVHrGFpKT60WyyWJd++DNvPfEJHIjfIVu17n0mbho+srNXHJGXV/arntNO7a9t8iXtl3bLnJEw15KcRqwUERWAF8Cs1R1jojcJiK3hZ4zG9gArAdeAwafTPbSZlnEFbo9ALhcVX8UkWeBxcCTRb0oNPg9AEBiqhMIVDmZjL4659Lz2bdzLz+s2siZbc72O45x1GtfrCUmIHQ5u8z7c8pM4mKo0+kC0h570/O2XVRe3SZV3QCcV8T6VwvdVqDcBs1LK8gBEalJXk9aVPXHUIgDIpJT3ItCA+NjAGLjU8r088nM2ErDBskF9xuk1Cczc2tZ3uqk/KpVU879XSvO6diS2ErxJFRN4Obn72DCXf/wPIvfXPlMXDNj5WY+S9vOP/u0QcT7EynUvqwl+1duJPtH74dxwL3toiIfqVdaQa4OfAUIoPnz60SkamhdxCxZupwmTRrTqFFDMjK20rv31dxwo/d7b2c8PYUZT08B4Mw2Z3P5n7qdksUY3PlMXLJow3Ymfrmesf0uJiHOn2n9p/k4XAHubRe5Ffi60yVuQaraqJiHcoFryj1NIcFgkCF3jmD2rDeJCQSYMHEqqalrI9mkcyr1vINAo2ZIYjUS7n6J7I/fRg/tJ77LzUhiEpWvG0pw6yaOTCpy5KjcufKZvDz2Gdq2u5BatWuwdNV8nn3yZd6a9G7E2x32/lcs3byTPYeO0mn0Rwy6pCnjF6/jaDCX26YtBuDc+jUZ0fnciGfJF0isRK325/LdPcfP1vKOK9tFvopbjvOGISLaQFmHLMrTn5Lb+R0BgGdvjvE7AtUfX+B3BMCNK4asf/ZKvyMA8IUDVwzptNu/HnZhOUczTvov73sa9Qu75jy7aYpTJ+u0Q6eNMVElWIH7yFaQjTFRJZp36hljTIWi1kM2xhg3WA/ZGGMcEbXT3owxpqKpuOXYCrIxJsrkVOCSbAXZGBNVbKee45Yc8ed0lceL/f0IvyPw99fdOAX2ny/d5ncEZ7hyUEa0sJ16xhjjCOshG2OMI6yHbIwxjghW4AtJWEE2xkQVm4dsjDGOsDFkY4xxhI0hG2OMI2zIwhhjHGFDFsYY44iKPMvCjcO2itG5UwdWr1rAd6kLGXqvPxdNPC25Lq++/SLTPv0XUz95g7639vKs7QdHT+a3fxzONXc/8bPHJn7wX8699i/sztrvWR6A2xe+wJ/mPsmtsx/nlg8e8azdhFvuodqL/6bqI68VrJMq1Ui85ymqPjmBxHuegsSqEc3w0IfL6fjSXHqO/6Rg3ciPU+kx9r9c+/on3PXeErIOZ0c0w/Fc+I64lAPyhizCXVzjbEEOBAKMevExuna7nubndaRPnx40a3am5zlycoI8//eX6f3bG+h/1UCuvfn3ND6rkSdtd+9wEa/cP+hn67fu2M0XK76jfp2anuQ43qS+jzK2y3DGd3vAszaPLpzLgZF/PWZdpS59CaYuY/+wmwmmLqPyVX0jmqH7OQ0Z3euiY9a1aVSHt2/pwL/7d+D0mlUYv3hdRDMU5sp3xJUc+XJPYHFNiQVZRC4SkaTQ7QQR+buIfCAiT4lI9UgGa31hS9LSNrFx4w9kZ2czbdoMunfrHMkmi7Rz+07WrMy7gu7BA4fYtG4TdevV8aTtVmc3oXrVxJ+tf3rCu9x1/dWIOHV9xogKrl2J7t93zLrYlhdzdNE8AI4umkdsy8hezPaChrVJSog/Zt3FjesSG8j7Gp2bXJNt+w5HNENhrnxHXMmRT0/gP9eU1kMeDxwM3X4RqA48FVr3egRzkZxSj83pmQX30zO2kJxcL5JNlqp+g3o0bX4Wq75O9S3Dx0u+oW6tGjRtlOJTAuW6ScO4ZeajtOzX0acMeQLVa6J7d+Wl2ruLQHV//mLIN33lZi45o65n7bnyHXElR76KPGRR2k69gKrmhG63UtXzQ7cXisjy4l4kIgOAAQASU51AoMrJJ/VZQmICT497lOceHMWB/QdLf0EEHDpylNfe/Yh/jhjsS/sAb/R8mH3bdpNYO4nrJg1jR9oWNn/5nW95juHjzpzXvlhLTEDocrZfvyhNPo3inXqrRKR/6PYKEWkFICJnAcXuvVDVMaraSlVblbUYZ2ZspWGD5IL7DVLqk5m5tUzvdbJiYmN4etyjzHn3Iz6evcCXDACbt+4gY/tOrr33Ka4Y/De27dxDn6HPsGN3lmcZ9m3bDcDBnVmsmbuU5BZneNb28XL37kaq1wJAqtciN2uPLzlmrNzMZ2nbebxrS0+HkVz5jriSI18QDXspiYg0FJGPRSRVRFaLyJAintNBRPaKyPLQ8uDJZC+tIN8K/FZE0oCzgS9EZAPwWuixiFmydDlNmjSmUaOGxMXF0bv31Xwwc14kmyzWgyOHsXHdJib/c6ov7ec76/RkPh33OHNG/405o//GabVrMPXpe6lTM8mT9uMSKhFfpXLB7TPaN+fHNemetF2UnOVfEN+uEwDx7TqRs+xzzzMs2rCdiV+u54XfX0hCnLezSF35jriSI185DlnkAP9PVc8G2gC3i8jZRTzvM1VtEVoePpnsJW5BqroXuDm0Y69x6Pnpqhrxs4sHg0GG3DmC2bPeJCYQYMLEqaSmro10sz9zXuvmXHXtFaxLTWPyR+MBGP3EGBb9d3HE2x76wgSWrl7Pnn37+d3ABxjcuwu/v6xtxNstTpU6SfQacxcAgdgYVs/4nA2ffuNJ2wkDhxP76/OQqtWp9twUDk+fyJFZb5E4eARx7a9Ad2zn4CuRnYY37P2vWLp5J3sOHaXT6I8YdElTxi9ex9FgLrdNy9sezq1fkxGdz41ojnyufEdcyZGvvIYsVHULsCV0e5+IfAukABHbiSSRHm+JjU/xfUCnRW3//qwubNFc/68Y8my3N/yOALhxxZD4S1r4HQGAaoOm+B3BGTlHM056zKdjg8vDrjkfp38UVnsi0ghYAJyjqlmF1ncA3gHSgUzgHlVdfQJxj2FH6hljosqJTGcrPAEhZIyqjjnuOVXJK7p3Fi7GIV8Dp6vqfhHpAkwHyjwJ2wqyMSaqnMih06HiO6a4x0UkjrxiPFlV3y3i9VmFbs8WkdEiUkdVd5xY6jxWkI0xUaW85hdL3pSZccC3qjqymOfUA7apqopIa/ImSuwsa5tWkI0xUaUcD/hoB9wArCx03MVw4JcAqvoq0AsYJCI5wCGgr57EjjkryMaYqFKOsywWAiXu9FPVl4CXyqVBrCAbY6KMi4dEh8sKsjEmqrh40qBwWUE2xkSVoLp4Ys3wnBIFefnODX5HAGDqlf/yOwJ3j/fvSL/Cjr413e8IdkBGlKrIJxc6JQqyMebUYWPIxhjjCBtDNsYYR+TakIUxxrjBesjGGOMIm2VhjDGOsCELY4xxhA1ZGGOMI6yHbIwxjqjIPeTSLnLqq86dOrB61QK+S13I0HtvP6VzxCUl8tsxf+HqT5+m+ydPUeeCJp60+9Dk/9Bx+Fh6PjG5YN3LsxZz7ZNv0vupKdz28nS2790f8RwJt9xDtRf/TdVHXitYJ1WqkXjPU1R9cgKJ9zwFiVUjnqMwF7YLFzK4lAMgqMGwF9c4W5ADgQCjXnyMrt2up/l5HenTpwfNmpX5yigVPkfrh28g4+NvmPHbocy8fDh712V60m73i5oxelD3Y9bddOn5/HvYdUy7rx/tz2nMmDlLIp7j6MK5HBj512PWVerSl2DqMvYPu5lg6jIqX9U34jnyubBduJDBpRz5VDXsxTUlFmQR+YuINPQqTGGtL2xJWtomNm78gezsbKZNm0H3bp1PyRxx1RKoe1FT1k/5BIDc7CDZWQc9afuCJikkJVY+Zl3VhPiC24eOZJd8wthyEly7Et2/75h1sS0v5uiivMvNH100j9iW7TxIkseF7cKFDC7lyJeLhr24prQe8iPA/0TkMxEZLCK/8CIUQHJKPTan/9QLTM/YQnJyPa+adypH1V/+giM793Hx8wPoOvdR2j5zK7EJlTzNcLx/zPyCzg++zuyv1jCoSxtfMgSq10T37gJA9+4iUL2mZ227sF24kMGlHPmitocMbAAakFeYLwBSRWSOiNwkItWKe5GIDBCRpSKyNDf3QDnGPTUFYmKo1bwRa9+Yz8zOI8g5eIRz/tzN10x3dG3L3If70+WCprz12QpfsxRw8AtmvJerGvbimtIKsqpqrqrOU9U/AsnAaOAK8op1cS8ao6qtVLVVIFClTMEyM7bSsEFywf0GKfXJzNxapvc6GS7kOLBlFwe37GLHsjQAvp/1JbWaN/I0Q3G6tGrK/BVpvrSdu3c3Ur0WAFK9FrlZezxr24XtwoUMLuXIpyfwn2tKK8jHDA+qaraqvq+q/YDTIxcLlixdTpMmjWnUqCFxcXH07n01H8ycF8kmnc1x+Me9HMjcRdKv6gNQ/5L/Y+/aDE8zFPb99p8K3ycrN9C4rndDBYXlLP+C+HadAIhv14mcZZ971rYL24ULGVzKkS+ouWEvriltHnKf4h5Q1YjuVQoGgwy5cwSzZ71JTCDAhIlTSU1dG8kmnc7x5QMTueQfg4iJi2XfD9v5/O4xnrQ7bMIclq7PYM/+w3R6YDyDulzEwtTv2bR9NwER6tesxv19OkY8R8LA4cT++jykanWqPTeFw9MncmTWWyQOHkFc+yvQHds5+MojEc+Rz4XtwoUMLuXI5+LYcLgk0uFj41Mq7k+nnI3/ReQLV2muHe/PDrjjuXDFkNqTv/U7gjlOztGMk560U6vamWHXnF371nkxSShsdqSeMSaqVOQeshVkY0xUcXF+cbisIBtjoor1kI0xxhEuzp4IlxVkY0xUcfGAj3BZQTbGRJWKPGTh7NnejDGmLMrzSD0RuUJE1ojIehEZVsTjlURkaujx/4lIo5PJbgXZGBNVyuvkQiISA7wMXAmcDfQTkbOPe9ofgd2q2gR4HnjqZLJbQTbGRJVyPLlQa2C9qm5Q1aPAW8DVxz3namBi6PbbwGUiUvaDTU7kt4lfCzDAMriTw4UMruRwIYMrOVzIUJbMwNJCy4BCj/UCxha6fwPw0nGvXwU0KHQ/DahT1jwVpYc8wO8AuJEB3MjhQgZwI4cLGcCNHC5kOCFa6MyUocWbk8QUo6IUZGOM8VoGUPiKSQ1C64p8jojEAtWBnWVt0AqyMcYUbQlwpog0FpF4oC/w/nHPeR+4KXS7F/BfDY1dlEVFmYfs658RIS5kADdyuJAB3MjhQgZwI4cLGcqNquaIyJ+BuUAMMF5VV4vIw8BSVX0fGAf8S0TWA7vIK9plFvHTbxpjjAmPDVkYY4wjrCAbY4wjnC7IpR226FGG8SKyXURW+dF+KENDEflYRFJFZLWIDPEpR2UR+VJEVoRy/N2PHKEsMSKyTERm+phhk4isFJHlIrLUpww1RORtEflORL4VkbY+ZGga+hnkL1kicqfXOaKBs2PIocMW1wKXA+nk7fHsp6qpHudoD+wH3lDVc7xsu1CG+kB9Vf1aRKoBXwE9fPhZCFBFVfeLSBywEBiiqou9zBHKcjfQCkhS1a5etx/KsAlopao7/Gg/lGEi8Jmqjg3NBEhUVe8uv/3zPDHkTQW7SFW/9ytHReVyDzmcwxYjTlUXkLf31DequkVVvw7d3gd8C6T4kENVdX/oblxo8fw3uog0AK4CxnrdtktEpDrQnrw9/ajqUT+LcchlQJoV47JxuSCnAJsL3U/HhyLkmtDZpFoC//Op/RgRWQ5sBz5SVT9yvAAMBfw+E7kC80TkKxHx4yi1xsCPwOuh4ZuxIlLFhxyF9QWm+JyhwnK5IJvjiEhV4B3gTlXN8iODqgZVtQV5Ry21FhFPh3FEpCuwXVW/8rLdYlyiqueTdzaw20PDW16KBc4HXlHVlsABwJd9LQChIZPuwL/9ylDRuVyQwzls8ZQRGrN9B5isqu/6nSf0p/HHwBUeN90O6B4av30LuFREJnmcAQBVzQj9ux14j7xhNi+lA+mF/kp5m7wC7Zcrga9VdZuPGSo0lwtyOIctnhJCO9PGAd+q6kgfc/xCRGqEbieQt8P1Oy8zqOpfVbWBqjYib5v4r6pe72UGABGpEtrBSmiYoBN5Z/7yjKpuBTaLSNPQqssAT3f0HqcfNlxxUpw9dLq4wxa9ziEiU4AOQB0RSQceUtVxHsdoR96p/1aGxm8BhqvqbI9z1AcmhvakB4BpqurbtDOfnQa8Fzr1bSzwpqrO8SHHHcDkUKdlA9Dfhwz5v5QuBwb60X60cHbamzHGnGpcHrIwxphTihVkY4xxhBVkY4xxhBVkY4xxhBVkY4xxhBVkY4xxhBVkY4xxxP8HBhsgf+LWQBYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "digit = X[0].reshape([8, 8])\n",
    "sns.heatmap(digit, annot=True);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "我们需要划分一个训练集与验证集，在训练集上训练 `Step 1`得到的机器学习模型，在验证集上预测，预测值会被处理为返回的损失值 `loss` 。首先我们要对原数据进行切分以得到训练集和验证集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "     X, y, test_size=0.33, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "然后我们需要在训练集`X_train`，`y_train`上对机器学习模型`ML_model`进行训练："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearSVC(C=1.0, class_weight=None, dual=True, fit_intercept=True,\n",
       "          intercept_scaling=1, loss='squared_hinge', max_iter=600,\n",
       "          multi_class='ovr', penalty='l2', random_state=42, tol=0.0001,\n",
       "          verbose=0)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ML_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "#### Step 3. 在验证集上得到相应的评价指标"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "为了保证我们的机器学习模型具有对未知数据的泛化能力，所以需要用未参与训练的验证集来评价之前得到的机器学习模型。\n",
    "\n",
    "所以，我们需要一个评价指标。这里我们选用最简单的 `accuracy_score`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "y_pred = ML_model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9444444444444444"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "我们注意到本示例的 `Step 2`与`Step 3`的训练与验证部分可能存在偏差。比如如果数据集划分得不均匀，可能会在验证集上过拟合。\n",
    "\n",
    "我们可以用**交叉验证**(Cross Validation, CV)的方法解决这个问题，交叉验证的原理如图所示："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "<img src=\"https://scikit-learn.org/stable/_images/grid_search_cross_validation.png\" width=500></img>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import StratifiedKFold # 采用分层抽样\n",
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "因为这只是一个简单的示例，所以采用3-Folds交叉验证以减少计算时间。\n",
    "\n",
    "在实践中建议使用5-Folds交叉验证或10-Folds交叉验证。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.92654424, 0.94490818, 0.96327212])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = cross_val_score(ML_model, X, y, cv=cv)\n",
    "scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "我们采用交叉验证的平均值作为最后的得分："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9449081803005009"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "score = scores.mean()\n",
    "score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "#### Step 4. 对评价指标进行处理，使其越小越好，返回loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "我们注意到，评价指标正确率的取值范围是0-1，且是越大越好的，所以我们需要将score处理为越小越好的loss："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.055091819699499056"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss = 1 - score\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "#### 用一个函数实现AutoML评价器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "metric = \"accuracy\"\n",
    "def evaluate(config: dict) -> float:\n",
    "    layered_dict = layering_config(config)\n",
    "    display(plot_layered_dict(layered_dict))  # 用于在jupyter notebook中可视化，实践中可以删除次行\n",
    "    AS_HP = layered_dict['classifier'].copy()\n",
    "    AS, HP = AS_HP.popitem()\n",
    "    ML_model = eval(AS)(**HP)\n",
    "    # 注意到： X, y, cv, metric 都是函数外的变量\n",
    "    scores = cross_val_score(ML_model, X, y, cv=cv, scoring=metric)\n",
    "    score = scores.mean()\n",
    "    print(f\"accuracy: {score:}\") # 用于在jupyter notebook中调试，实践中可以删除次行\n",
    "    return 1 - score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n",
       " -->\n",
       "<!-- Title: Layered Dict Pages: 1 -->\n",
       "<svg width=\"655pt\" height=\"188pt\"\n",
       " viewBox=\"0.00 0.00 655.49 188.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 184)\">\n",
       "<title>Layered Dict</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-184 651.495,-184 651.495,4 -4,4\"/>\n",
       "<!-- classifier -->\n",
       "<g id=\"node1\" class=\"node\"><title>classifier</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"290.5,-180 225.5,-180 225.5,-144 290.5,-144 290.5,-180\"/>\n",
       "<text text-anchor=\"middle\" x=\"258\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\">classifier</text>\n",
       "</g>\n",
       "<!-- LinearSVC -->\n",
       "<g id=\"node2\" class=\"node\"><title>LinearSVC</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"297,-108 219,-108 219,-72 297,-72 297,-108\"/>\n",
       "<text text-anchor=\"middle\" x=\"258\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\">LinearSVC</text>\n",
       "</g>\n",
       "<!-- classifier&#45;&gt;LinearSVC -->\n",
       "<g id=\"edge8\" class=\"edge\"><title>classifier&#45;&gt;LinearSVC</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M258,-143.697C258,-135.983 258,-126.712 258,-118.112\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"261.5,-118.104 258,-108.104 254.5,-118.104 261.5,-118.104\"/>\n",
       "</g>\n",
       "<!-- C -->\n",
       "<g id=\"node3\" class=\"node\"><title>C</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">C</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;C -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>LinearSVC&#45;&gt;C</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M218.892,-80.5091C179.065,-71.3674 115.83,-55.4811 63,-36 61.1518,-35.3185 59.2689,-34.578 57.382,-33.8001\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"58.6185,-30.5205 48.0555,-29.6992 55.8008,-36.9284 58.6185,-30.5205\"/>\n",
       "</g>\n",
       "<!-- dual -->\n",
       "<g id=\"node4\" class=\"node\"><title>dual</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"99\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"99\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">dual</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;dual -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>LinearSVC&#45;&gt;dual</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M218.796,-72.5127C194.562,-62.2205 162.869,-48.58 135,-36 133.205,-35.1895 131.363,-34.3489 129.507,-33.4947\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"130.807,-30.2392 120.264,-29.1893 127.851,-36.5845 130.807,-30.2392\"/>\n",
       "</g>\n",
       "<!-- loss -->\n",
       "<g id=\"node5\" class=\"node\"><title>loss</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"171\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"171\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">loss</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;loss -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>LinearSVC&#45;&gt;loss</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M236.494,-71.6966C224.111,-61.7332 208.501,-49.1733 195.627,-38.8148\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"197.609,-35.917 187.623,-32.3751 193.22,-41.3708 197.609,-35.917\"/>\n",
       "</g>\n",
       "<!-- max_iter -->\n",
       "<g id=\"node6\" class=\"node\"><title>max_iter</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"258\" cy=\"-18\" rx=\"42.4939\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"258\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">max_iter</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;max_iter -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>LinearSVC&#45;&gt;max_iter</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M258,-71.6966C258,-63.9827 258,-54.7125 258,-46.1124\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"261.5,-46.1043 258,-36.1043 254.5,-46.1044 261.5,-46.1043\"/>\n",
       "</g>\n",
       "<!-- multi_class -->\n",
       "<g id=\"node7\" class=\"node\"><title>multi_class</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"370\" cy=\"-18\" rx=\"51.1914\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"370\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">multi_class</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;multi_class -->\n",
       "<g id=\"edge5\" class=\"edge\"><title>LinearSVC&#45;&gt;multi_class</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M285.398,-71.8761C301.026,-62.1089 320.716,-49.8028 337.167,-39.5207\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"339.302,-42.3137 345.927,-34.0456 335.592,-36.3777 339.302,-42.3137\"/>\n",
       "</g>\n",
       "<!-- penalty -->\n",
       "<g id=\"node8\" class=\"node\"><title>penalty</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"476\" cy=\"-18\" rx=\"36.2938\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"476\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">penalty</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;penalty -->\n",
       "<g id=\"edge6\" class=\"edge\"><title>LinearSVC&#45;&gt;penalty</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M297.134,-77.6104C332.125,-67.268 384.718,-51.3311 430,-36 432.722,-35.0785 435.525,-34.1045 438.342,-33.1079\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"439.646,-36.3582 447.868,-29.6756 437.274,-29.7726 439.646,-36.3582\"/>\n",
       "</g>\n",
       "<!-- random_state -->\n",
       "<g id=\"node9\" class=\"node\"><title>random_state</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"589\" cy=\"-18\" rx=\"58.4896\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"589\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\">random_state</text>\n",
       "</g>\n",
       "<!-- LinearSVC&#45;&gt;random_state -->\n",
       "<g id=\"edge7\" class=\"edge\"><title>LinearSVC&#45;&gt;random_state</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M297.331,-81.6608C348.854,-71.9122 441.942,-53.8381 521,-36 525.722,-34.9345 530.619,-33.7896 535.522,-32.6158\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"536.649,-35.9438 545.54,-30.1818 534.997,-29.1417 536.649,-35.9438\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f0fd8a9f128>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.9449081803005009\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.055091819699499056"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(CS.get_default_configuration())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "#### 用一个类实现AutoML评价器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "我们注意到，`evaluate`函数的 X, y, cv, metric 都是函数外的变量，不利于管理，所以实践中我们一般采用定义**类**来实现评价器。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "首先我们定义一个类，这个类需要根据 训练数据、评价指标和交叉验证方法 等条件进行初始化："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "default_cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=0)\n",
    "\n",
    "class Evaluator():\n",
    "    def __init__(self, \n",
    "        X, y, \n",
    "        metric=\"accuracy\", \n",
    "        cv=default_cv):\n",
    "        # 初始化\n",
    "        self.X = X\n",
    "        self.y = y\n",
    "        self.metric = metric\n",
    "        self.cv = cv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "然后我们需要实现`__call__`魔法方法，在该方法中实现整个评价过程："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "default_cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=0)\n",
    "\n",
    "class Evaluator():\n",
    "    def __init__(self, \n",
    "        X, y, \n",
    "        metric=\"accuracy\", \n",
    "        cv=default_cv):\n",
    "        # 初始化\n",
    "        self.X = X\n",
    "        self.y = y\n",
    "        self.metric = metric\n",
    "        self.cv = cv\n",
    "    \n",
    "    def __call__(self, config: dict) -> float:\n",
    "        layered_dict = layering_config(config)\n",
    "        AS_HP = layered_dict['classifier'].copy()\n",
    "        AS, HP = AS_HP.popitem()\n",
    "        ML_model = eval(AS)(**HP)\n",
    "        scores = cross_val_score(ML_model, self.X, self.y, cv=self.cv, scoring=self.metric)\n",
    "        score = scores.mean()\n",
    "        return 1 - score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "实例化一个评价器对象："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "evaluator = Evaluator(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "用配置空间的一个采样样本测试这个AutoML评价器："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.017250973845297835"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluator(CS.sample_configuration())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "### 根据上述知识实现一个简单的AutoML系统"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "**配置空间** 、**评价器**我们都有了，UltraOpt已经实现了成熟的**优化器**，所以我们只需要用`ultraopt.fmin`函数将这些组件串起来"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "from ultraopt import fmin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|██████████| 40/40 [00:14<00:00,  2.67trial/s, best loss: 0.012]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "+----------------------------------------------------------------------------+\n",
       "| HyperParameters                                     | Optimal Value        |\n",
       "+-----------------------------------------------------+----------------------+\n",
       "| classifier:__choice__                               | KNeighborsClassifier |\n",
       "| classifier:KNeighborsClassifier:n_neighbors         | 4                    |\n",
       "| classifier:KNeighborsClassifier:p                   | 2:int                |\n",
       "| classifier:KNeighborsClassifier:weights             | distance             |\n",
       "| classifier:LinearSVC:C                              | -                    |\n",
       "| classifier:LinearSVC:dual                           | -                    |\n",
       "| classifier:LinearSVC:loss                           | -                    |\n",
       "| classifier:LinearSVC:max_iter                       | -                    |\n",
       "| classifier:LinearSVC:multi_class                    | -                    |\n",
       "| classifier:LinearSVC:penalty                        | -                    |\n",
       "| classifier:LinearSVC:random_state                   | -                    |\n",
       "| classifier:RandomForestClassifier:bootstrap         | -                    |\n",
       "| classifier:RandomForestClassifier:criterion         | -                    |\n",
       "| classifier:RandomForestClassifier:max_features      | -                    |\n",
       "| classifier:RandomForestClassifier:min_samples_leaf  | -                    |\n",
       "| classifier:RandomForestClassifier:min_samples_split | -                    |\n",
       "| classifier:RandomForestClassifier:n_estimators      | -                    |\n",
       "| classifier:RandomForestClassifier:random_state      | -                    |\n",
       "+-----------------------------------------------------+----------------------+\n",
       "| Optimal Loss                                        | 0.0117               |\n",
       "+-----------------------------------------------------+----------------------+\n",
       "| Num Configs                                         | 40                   |\n",
       "+-----------------------------------------------------+----------------------+"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = fmin(evaluator, HDL, optimizer=\"ETPE\", n_iterations=40)\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "我们可以对AutoML得到的结果进行数据分析：\n",
    "\n",
    "首先， 我们可以绘制拟合曲线："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEYCAYAAACdnstHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de5xcZZ3n8c+3u7o6fUknIU2aSxISLgtEBGLCzYkKAhrckeAYR1ARRxxeqzI6yzorOsqwLK6DrqKueEFwREQDg6JRUOSSRlFuCQkkIQRDQkICJAY6JH1Jqi+//eOcSk4q1d11qrtuXb/363VedS7PeepXp7vr6ec5z3kemRnOOedcPmpKHYBzzrnK5YWIc865vHkh4pxzLm9eiDjnnMubFyLOOefy5oWIc865vHkh4pwbkqQZkkxSotSxuPLjhYiraJI+IGmppE5JL0v6raR5pY6rWkm6WtJPSh2HKx4vRFzFknQF8A3g/wBtwHTgO8CCUsYV5f+9u7HOCxFXkSRNAK4BPmlmvzCzLjPrNbNfm9m/hGnqJX1D0kvh8g1J9eGxMyVtlvQ/JG0LazH/EB47TdIrkmoj7/ceSU+H6zWSrpT0vKRXJd0h6aDwWLrp51JJm4AHw/0flrQxTP9FSS9IOidGfpdI2iRpu6R/jcRVK+nz4bm7JC2TNC08dpyk+yS9JmmtpL8f4nq2S/qypMcl7ZT0q3QMWdIeJmlxmO86Sf8Y7p8PfB54f1gzfCqvH66rKF6IuEp1BjAOuGuINP8KnA6cDJwEnAp8IXL8EGACcDhwKXCDpElm9hjQBbw9kvYDwE/D9X8CLgDeBhwGdAA3ZLz324DjgXdKmkVQQ/ogcGjkPdNyyW8ecCxwNnCVpOPD/VcAFwHvAlqAjwLdkpqA+8KYpwAXAt8JYxnMh8PzDwX6gG8Nkm4RsDmMdSHwfyS93cx+R1ArvN3Mms3spCHey40VZuaLLxW3EHwhvzJMmueBd0W23wm8EK6fCfQAicjxbcDp4fq1wA/D9fEEhcoR4fYa4OzIeYcCvUACmAEYcGTk+FXAzyLbjUAKOCdGflMjxx8HLgzX1wILsnz29wN/zNj3feDfBrlW7cC/R7ZnhTHWRmJIANOAfmB8JO2XgR+F61cDPyn174cvxVu8vdZVqleBVkkJM+sbJM1hwMbI9sZw3948Ms7tBprD9Z8Cf5b0ceDvgCfNLJ3XEcBdkgYi5/YT3JdJezEjjr3bZtYt6dXI8Vzye2WQOKcRFJaZjgBOk7Qjsi8B3JolbbaYNwJ1QGtGmsOA18xsV0bauUPk68Ywb85yleoRYA9BM9BgXiL4Mk2bHu4blpk9Q/DleB77N2VB8GV7nplNjCzjzGxLNIvI+svA1PSGpAZgcsz8BvMicNQg+x/KyLPZzD4+RF7TIuvTCWpD2zPSvAQcJGl8Rtp0rD4seJXxQsRVJDN7naCZ6AZJF0hqlFQn6TxJXwmT/Qz4gqSDJbWG6eN0P/0p8GngrcB/RvZ/D/iSpCMAwvyH6hF2J/BuSW+WlCRo8tEI8ou6Cfjfko5R4ERJk4HfAP9F0sXhdamTdErkXko2H5I0S1IjQaeFO82sP5rAzF4E/gx8WdI4SScS3E9KX9etwAxJ/t1SJfwH7SqWmX2N4MbyF4C/Evz3fTnwyzDJtcBS4GlgJfBkuC9XPyO42f2gmUX/I/8msBj4vaRdwKPAaUPEuZrg5vkiglpJJ8H9lz355Jfh68AdwO+BncDNQEPY3PQOghvqLxE0h10H1A+R163Aj8K044BPDZLuIoL7JC8RdGz4NzO7PzyWLmxflfRkjp/BVTCZee3TuWKS1AzsAI4xsw2ljgeCLr4EN8RvKnUsrrJ4TcS5IpD07rDJrQn4vwQ1oxdKG5VzI+eFiHPFsYCg+ecl4BiCLrreDOAqnjdnOeecy5vXRJxzzuWt6h42bG1ttRkzZgybrquri6ampsIHFFO5xgXlG5vHFV+5xuZxxTOacS1btmy7mR18wIFSPzJf7GXOnDmWiyVLluSUrtjKNS6z8o3N44qvXGPzuOIZzbiApZblO9Wbs5xzzuXNCxHnnHN580LEOedc3rwQcc45lzcvRJxzzuXNCxHnnHN580LEOedc3rwQcc45l7eqe2J9pHZ0p9iwvYudPb20NNQxs7WJiY3JUTl3JHmXUjruzj19LN/UsV/cI/1MQ51fyrxHcnyo61VqpYqtkH9XrrC8JhLDju4Uyzd1kOobYFJjklTfAMs3dbCjOzXic0eSdylF466t0X5xj/QzDXV+KfMeyfGhrleplSq2Qv5ducLzQiSGDdu7aEwmuGPpZpas/SuNyQSNyQQbtnflfG5jMoGkA84d7ni5isYN7Bf3SD/TUOeXMu+RHB/qepVaqWIbyc+yUv9uxhIvRGLY2dNLQ10tf/zLX3lq8w4AGupq2dnTm/O5UdFzhzue/o8r3cxQLv9pDRX3cJ+p0HkPdc1GkvdIjo/0mhRSqWIbyfuW8/WsFkUrRCTNl7RW0jpJV2Y5Xi/p9vD4Y5JmZByfLqlT0mci+16QtFLSCklLC/0ZWhrq6OntZ1xdLT2pfgB6evtpaajL+dyo6LlDHS/nJpCh4h7uMxcy7+Gu2UjyHsnxkV6TQipVbCN533K+ntWiKIWIpFrgBuA8YBZwkaRZGckuBTrM7GjgeuC6jONfB36bJfuzzOxkM5s7ymEfYGZrE92pPuoTNfT09tOd6qM71cfM1uGHWk6f253qw8wOOHeo4+XcBBKNG9gv7uE+c5y8M88fLu/hrtlI8h7J8aGuV6mVKraR/J6M9HfMjVyxaiKnAuvMbL2ZpYBFBNOFRi0AbgnX7wTOliQASRcAG4DVRYo3q4mNSWZPn0RjspbO3b0kEzXMnj4pp54g6XOTiRo6ulMHnDvU8XKuskfj7h+w/eIe7jPHyTvz/OHyHu6ajSTvkRwf6nqVWqliG8nvyUh/x9zIFWV6XEkLgflm9rFw+2LgNDO7PJJmVZhmc7j9PHAasBu4DzgX+AzQaWb/N0yzAegADPi+md04yPtfBlwG0NbWNmfRokXDxtzZ2Ulzc3PWY/9v+W62dg1w7bzGHD79yPWk+hkAagS9u7upG9fIgAX/ATQka4c7vWiGumbFVgnXrJyuV6Zyjc3jimc04zrrrLOWZWvxqYTnRK4GrjezzrBiEjXPzLZImgLcJ+lZM/tDZqKwcLkRYO7cuXbmmWcO+6bt7e0Mlm7xthW8sv61QY+PtnT7fmMywWvPr+Cgo06mO9VXdv9xDXXNiq0Srlk5Xa9M5RqbxxVPMeIqVnPWFmBaZHtquC9rGkkJYALwKkFt5CuSXgD+Gfi8pMsBzGxL+LoNuIug2azgmusTdIXtxsVQzk0g5cqvmXPFUayayBPAMZJmEhQWFwIfyEizGLgEeARYCDwYTsn4lnQCSVcTNGd9W1ITUGNmu8L1dwDXFPyTAE31Cbr39A+fcBQFX4pJXl+fYPb0SaOa91h94reQ18w5FyhKTcTM+oDLgXuBNcAdZrZa0jWSzg+T3QxMlrQOuAI4oBtwhjbgYUlPAY8Dd5vZ7wrzCfbXlKwl1T9Aqm+gGG9XUP7Er3NuJIp2T8TM7gHuydh3VWR9N/C+YfK4OrK+HjhpdKPMTVN9cNm69vSRTFT2f+zZusKm98+eXtmfzTlXeJVwY73s7C1EUn1MaqrsL9qdPb1Mymi6aqirpaNINZGx2pTmXLXwYU/y0JRM10SKe1+kEEr5xK83pTlX+bwQyUNTffCcQeee4vXQKpRSPvHrg+c5V/m8EMlDc+SeSKUr5RO/5fwkvnMuN35PJA/peyLdRXxWpJDSXWGLLd2Ulr6ZDz54nnOVxmsieUjfE+kcA/dESskHz3Ou8nkhkof0PZGx0JxVSj54nnOVz5uz8hDt4utGplRNac650eE1kTzUJ2qorZHXRJxzVc8LkTxIoilZOyaeE3HOuZHwQiRPzfWJMfGciHPOjYQXInlqqk+MmS6+zjmXLy9E8tRYn/Auvs65queFSJ6a62v9xrpzrup5IZKnpmTCCxHnXNXz50Ty1FTkKXKdc8XnUxUMz2sieWqq9y6+zo1lPlVBbrwQyVOTd/F1bkzzqQpy44VInpqTCVJ9A/T2V/486865A/lUBbkpWiEiab6ktZLWSboyy/F6SbeHxx+TNCPj+HRJnZI+k2uehdSYHg7em7ScG5NKOetnJSlKISKpFrgBOA+YBVwkaVZGskuBDjM7GrgeuC7j+NeB38bMs2Ca07Mb+s1158Ykn6ogN8WqiZwKrDOz9WaWAhYBCzLSLABuCdfvBM6WJABJFwAbgNUx8yyYvRNT+X0R58Ykn6ogN8Xq4ns48GJkezNw2mBpzKxP0uvAZEm7gc8C5wKfyZZ+iDwBkHQZcBlAW1sb7e3twwbc2dk5ZLrn/xoUHn945HG2TKwdNN1oGy6uUirX2Dyu+Mo1tlLG9TqwYn32Y9V8vSrhOZGrgevNrDOsmMRmZjcCNwLMnTvXzjzzzGHPaW9vZ6h0jRte4/plj3DsG05i3jGtecWVj+HiKqVyjc3jiq9cY/O44ilGXMUqRLYA0yLbU8N92dJslpQAJgCvEtQuFkr6CjARGAhrJ8tyyLNg0rMbejdf51w1K1Yh8gRwjKSZBF/0FwIfyEizGLgEeARYCDxoZga8JZ1A0tVAp5l9OyxohsuzYJrT90T8xrpzrorlfGNd0vskjQ/XvyDpF5LelMu5ZtYHXA7cC6wB7jCz1ZKukXR+mOxmgnsg64ArgCG77A6WZ66fZ6Qak+EUuV4Tcc5VsTg1kS+a2X9KmgecA3wV+C6D3MzOZGb3APdk7Lsqsr4beN8weVw9XJ7Fkq6J+HDwzrlqFqeLb/rb8r8CN5rZ3UDV9nUbV1dDjbw5yzlX3eIUIlskfR94P3CPpPqY548pwTzrPn6Wc666xSkE/p7g/sM7zWwHcBDwLwWJqkI01fucIs656pbzPREz6wZ+Edl+GXi5EEFVCh8O3jlX7XIuRMLmq/cCM6Lnmdk1ox9WZWj2iamcc1UuTu+sXxE8+b8M2FOYcCpLo0+R65yrcnEKkalmNr9gkVSgpvoEW3b0lDoM55wrmTg31v8s6Y0Fi6QCNdfXehdf51xVi1MTmQd8RNIGguYsAWZmJxYksgrQ6L2znHNVLk4hcl7BoqhQzT7PunOuyuXcnGVmGwlG0X13uEwM91WtpmSC3b0D9Pk86865KhVnAMZPA7cBU8LlJ5L+qVCBVYL0cPDdvf6siHOuOsVpzroUOM3MugAkXUcwbPv/K0RglSA9RW7Xnj5axtWVOBrnnCu+OL2zxL5BGAnX85tqcIyIFiLOOVeN4tRE/gN4TNJd4fYFBHOAVK3msDnLhz5xzlWrOGNnfV3SQ8DfhLv+wcyWFyasyuATUznnql2s6XHNbBnBsCeO6MRUXog456rTsIWIpIfNbJ6kXYBFDxE8bNhSsOjK3N57Iv7UunOuSg1biJjZvPB1fOHDqSxNfk/EOVfl4jwncl0u+4Y4f76ktZLWSboyy/F6SbeHxx+TNCPcf6qkFeHylKT3RM55QdLK8NjSXGMZLU1+T8Q5V+XidPE9N8u+nIZCkVQL3BCmnwVcJGlWRrJLgQ4zOxq4HkgXUKuAuWZ2MjAf+L6kaA3qLDM72czm5v5RRkdjshbJCxHnXPUathCR9HFJK4FjJT0dWTYAK3N8n1OBdWa23sxSwCJgQUaaBcAt4fqdwNmSZGbdZpb+lh7H/vdlSio9z3pXypuznHPVSWZDfydLmgBMAr4MRJuhdpnZazm9ibQQmG9mHwu3LyZ4+v3ySJpVYZrN4fbzYZrtkk4DfggcAVxsZneFaTYAHQQFy/fN7MZB3v8y4DKAtra2OYsWLRo25s7OTpqbm4dN989Lujnx4Fo+ekL9sGlHQ65xlUK5xuZxxVeusXlc8YxmXGedddayrC0+ZpbzQlCYnAq8Nb3keN5C4KbI9sXAtzPSrCKY+Cq9/TzQmpHmeOBxYFy4fXj4OgV4Kpd45syZY7lYsmRJTunO+uoS++Rty3JKOxpyjasUyjU2jyu+co3N44pnNOMCllqW79Q4N9Y/BvwBuBf4X+Hr1TmevgWYFtmeGu7Lmia85zEBeDWawMzWAJ3ACeH2lvB1G3AXQQFXVE0+p4hzrorFubH+aeAUYKOZnQXMBnbkeO4TwDGSZkpKAhcCizPSLAYuCdcXAg+amYXnJAAkHQEcB7wgqUnS+HB/E/AOgtpMUTXV1475eyI7ulMs39TBQ2u3sXxTBzu6U6UOyTlXJuIUIrvNbDcE3XHN7Fng2FxOtODG+OUEtZc1wB1mtlrSNZLOD5PdDEyWtA64gn33X+YBT0laQVDb+ISZbQfagIclPUXQxHW3mf0uxucZFU3JsV0TSRcgqb4BJjUmSfUNeEHinNsrzrAnmyVNBH4J3CepA8h5Uiozuwe4J2PfVZH13cD7spx3K3Brlv3rgZNyjr5Axnpz1obtXTQmE3vHCUu/btjexezpyVKG5pwrAzkVIpIEfMrMdgBXS1pCcM+i6P/5l5um+rHdxXdnTy+TGvcvLBrqaunwmohzjhwLkfDexD3AG8PthwoaVQVpStaO6ZpIS0MdPb39e2sgAD29/bQ0+CRczrl490SelHRKwSKpUE31CbpT/QwMlM0zkKNqZmsT3ak+ulN9mNne9ZmtTaUOzTlXBuLcEzkN+JCkF4Au9o3ie2IhAqsUzZGRfMePwSlyJzYmmT19Ehu2d9HRnaKloY5jD5nExEa/H1IqO7pTbNjexc6eXloa6pjZ2uQ/jxHw6zkycQqRdxYsigqWHg6+O9U/JgsRSBck/kdVDtK95RqTCSY1Junp7Wf5pg5mT/eCPR9+PUcuTnPWJuAtwCVmtpFgqJG2gkRVQdLDwfvEVK4Yor3lJO1d37C9q9ShVSS/niMXpxD5DnAGcFG4vYtgZN6q5sPBu2La2dNLQ13tfvsa6mrZ2dNboogqm1/PkYtTiJxmZp8EdgOYWQdQ9fW9vbMb+sRUrgjSveWivLdc/vx6jlycQqQ3nBfEACQdDAwUJKoKsm92Q6+JuMLz3nKjy6/nyMUpRL5FMOzIFElfAh4mGB6+qvk8666Y0r3lkokaOrpTJBM1fhN4BPx6jlzOvbPM7DZJy4CzCbr3XhCOqlvV0l18/ca6KxbvLTe6/HqOTM6FiKTrzOyzwLNZ9lWtvV18/Z6Ic64KFWWO9bGssc67+DrnqtewNRFJHwc+ARwp6enIofHAnwoVWKWoqRGNY3z8LOecG0wuzVk/BX7LCOZYH+vG+ki+zjk3mGELETN7HXidfQ8ZugxjfSRf55wbTC7NWQ+b2TxJuwifEUkfIhiAsaVg0VWIsT4xlXPODSaXmsi88HV84cOpTE31Cb+x7pyrSnF6Z7lBNIdzijjnXLUpWiEiab6ktZLWSboyy/F6SbeHxx+TNCPcf6qkFeHylKT35JpnsXjvLOdctSpKIRKOuXUDwXMls4CLJM3KSHYp0GFmRwPXA9eF+1cBc83sZGA+8H1JiRzzLIpmb85yzlWpnAsRSW+XdLOkr0n6B0lzJNXnePqpwDozW29mKWARsCAjzQLglnD9TuBsSTKzbjNLf0OPY9/N/VzyLIomb85yzlWpODWRHwK/Bh4FjgSuAlbneO7hwIuR7c3hvqxpwkLjdWAygKTTJK0GVgL/LTyeS55F0ZSspSscBdQ556pJnOlxN5rZL8P1/yxEMIMxs8eAN0g6HrhF0m/jnC/pMuAygLa2Ntrb24c9p7OzM6d0AK9sSWEG9z7QzriE4oQWW5y4iq1cY/O44ivX2DyueIoSl5nltAD/G/jvgHI9J3LuGcC9ke3PAZ/LSHMvcEa4ngC2Z3sv4EFgbi55ZlvmzJljuViyZElO6czMbn3kBTvis7+xra/35HxOvuLEVWzlGpvHFV+5xuZxxTOacQFLLct3apzmrFnAx4GXJd0t6UuS3pfjuU8Ax0iaKSkJXAgszkizGLgkXF8IPGhmFp6TAJB0BHAc8EKOeRZF8945Rfy+iHOuusSZT+S9AJIaCAqUNwKnkUPTlpn1SbqcoLZRC/zQzFZLuoagdFsM3AzcKmkd8BpBoQAwD7hSUi/BTIqfMLPtYSwH5Jnr5xlNjUmf3dA5V53i3BMBwMx6gGXhEue8e4B7MvZdFVnfDRxQszGzW4Fbc82zFHxiKudctfIn1kfB3ompfIpc51yV8UJkFDTVpyem8nsizrnqklMhosC0QgdTqdI1Eb8n4pyrNjkVImH3rpLfeyhXXog456pVnOasJyWdUrBIKlhTMl2IeHOWc666xOmddRrwIUkvAF3sm5TqxEIEVklqa8S4uhq6/Ma6c67KxClE3lmwKMYAH8nXOVeN4jRnbQLeAlxiZhsJRtNtK0hUFaipPkG3FyLOuSoTpxD5DsF4VReF27sI5vNwQGMy4V18nXNVJ9Y9ETN7k6TlAGbWEY5Z5YDmep/d0DlXfeLURHrD2QQNQNLBBGNZOYLmLL+x7pyrNnEKkW8BdwFTJH0JeBj4ckGiqkBN9QmviTjnqk6cUXxvk7QMOJuge+8FZramYJFVmKZkrT8n4pyrOjkXIpKuM7PPAs9m2Vf1vCbinKtGcZqzzs2y77zRCqTSNYf3RMznWXfOVZFhayKSPg58AjhS0tORQ+OBPxUqsErTmEwwYLC7d4CGcJIq55wb63JpznoX8LfAWuDdkf27zOy1gkRVgZr3Dgff54WIcw6AHd0pNmzvYmdPLy0NdcxsbWJi49h6MiKX5qyjgF6CQmQnwUOGuwAkHVS40CqLj+TrnIva0Z1i+aYOUn0DTGpMkuobYPmmDnZ0p0od2qjKpSbyPeABYCbBlLiKHDPgyALEVXH2FiL+rIhzDtiwvYvGZILGcJTv9OuG7V3Mnj52aiPD1kTM7FtmdjzwH2Z2pJnNjCxegIR8OHjnXNTOnl4a6vZv2m6oq2VnT2+JIiqMnHtnmdnHJU2SdKqkt6aXXM+XNF/SWknrJF2Z5Xi9pNvD449JmhHuP1fSMkkrw9e3R85pD/NcES5Tco1ntKWnyPXmLOccQEtDHT29+/9T2dPbT0tDXYkiKow4z4l8DPg0MBVYAZwOPAK8fajzwnNrCQZrPBfYDDwhabGZPRNJdinQYWZHS7oQuA54P7AdeLeZvSTpBOBe4PDIeR80s6W5fo5CafbmLOdcxMzWJpZv6gCCGkhPbz/dqT6OPWRSiSMbXXGeE/k0cAqw0czOAmYDO3I891RgnZmtN7MUsAhYkJFmAXBLuH4ncLYkmdlyM3sp3L8aaJBUHyPuovAb6865qImNSWZPn0QyUUNHd4pkoobZ0yeNud5ZyvXhOElPmNkpklYQjOi7R9JqM3tDDucuBOab2cfC7YvDPC6PpFkVptkcbj8fptmekc9/M7Nzwu12YDLQD/wcuNayfCBJlwGXAbS1tc1ZtGjRsJ+3s7OT5ubmYdOldfUan3ygm4uOS/LOGYWrrsaNq5jKNTaPK75yjc3jimc04zrrrLOWmdncAw6YWU4LweCLE4GrgT8AvwLuyfHchcBNke2LgW9npFkFTI1sPw+0RrbfEO47KrLv8PB1PPB74MPDxTJnzhzLxZIlS3JKl9bb129HfPY39s37n4t1Xlxx4yqmco3N44qvXGPzuOIZzbiApZblOzXOAIzvCVevlrQEmAD8LsfTtwDTIttTw33Z0myWlAjzfxVA0lSCQuzDZvZ8JKYt4esuST8laDb7ca6faTQlamuoT/g868656hLnnsheZvaQmS224P5GLp4AjpE0M5zI6kJgcUaaxcAl4fpC4EEzM0kTgbuBK81s7zArkhKSWsP1OoKn6lfl83lGiw/C6JyrNnkVInGZWR9wOUHPqjXAHWa2WtI1ks4Pk90MTJa0DrgCSHcDvhw4GrgqoytvPXBvOJ7XCoKazA+K8XkG01Tvw8E756pLnOlxR8TM7gHuydh3VWR9N/C+LOddC1w7SLZzRjPGkWpKek3EOVddYtdEJDWFz324DM0+Ra5zrsoMW4hIqpH0AUl3S9pGMCnVy5KekfRVSUcXPszK0FifoNObs5xzVSSXmsgSgpF8PwccYmbTzGwKMA94FLhO0ocKGGPFaK6v9eYs51xVyeWeyDlmdsCIYRbMJfJz4Odh76iq15RM0O2FiHOuiuQyim8vgKRvStJQaapdU32CTi9EnHNVJM6N9V3AYklNAJLeKcmnx41oqq+lK9Xv86w756pGnCfWvyDpA0C7pBTQyb5nORxBTaR/wNjTN8C4Ou/A5pwb++IMBX828I9AF3Ao8FEzW1uowCpRc2QkXy9EnHPVIE5z1r8CXzSzMwmGJbk9OkGU2zf9pT+17pyrFnGas94eWV8p6TyC3llvLkRglag5nN3Qb64756pFLg8bDtYj62Xg7KHSVJv0xFTd/tS6c65K5PSwoaR/kjQ9ujMcjfcMSbewb/TdqpZuzvKaiHOuWuTSnDUf+CjwM0kzCabEHQfUEkwE9Q0zW164ECvHvhvrfk/EOVcdcilErjOzT0v6EdALtAI9Zpbr/OpVoym8J+JDnzjnqkUuzVlvDV//aGa9ZvayFyDZ7a2J+D0R51yVyKUQeUDSI8Ahkj4qaY6k+kIHVon2dfH1QsQ5Vx2Gbc4ys89IOopgNN+ZwPnAG8Kn1leZ2fsLHGPFSCZqSNbW+HDwzrmqkdNzImb2vKRzzOy59D5JzcAJBYusQjXV13oXX+dc1YgzPe7GcOysGRnnPTqqEVW4xqSP5Oucqx5xhj35FbAA6CMYPyu95ETSfElrJa2TdMDAjZLqJd0eHn9M0oxw/7mSlklaGb6+PXLOnHD/OknfKoeHHpvrfZ5151z1iFMTmWpm8/N5k3BO9huAc4HNwBOSFpvZM5FklwIdZna0pAuB64D3A9uBd5vZS5JOAO4FDg/P+S7BoJCPAfcQPNPy23xiHC1N9bX+nIhzrmrEqYn8WdIb83yfU4F1ZrbezFLAIoJaTdQC4JZw/U7gbEkys+Vm9lK4fzXQENZaDgVazOxRCybw+DFwQZ7xjZqm+oR38XXOVY04NZF5wEckbQD2AALMzE7M4dzDgf1EyjYAABYuSURBVBcj25uB0wZLY2Z9kl4HJhPURNLeCzxpZnskHR7mE83zcLKQdBlwGUBbWxvt7e3DBtzZ2ZlTukzdO3ezrWsgr3NzkW9cxVCusXlc8WWLrX/ASPUN0G9GrUQyUUNtTXFbkMv1mlVzXHEKkfMKFkUOJL2BoInrHXHPNbMbgRsB5s6da2eeeeaw57S3t5NLuky/3vYUr6x/Na9zc5FvXMVQrrF5XPFlxrajO8XyTR0clEzQUFdLT28/3ak+Tp4+iYmNyZLFVS6qOa44Q8FvHMH7bAGmRbanhvuypdksKQFMAF4FkDQVuAv4sJk9H0k/dZg8i665vtabs9yYs2F7F43JxN4HatOvG7Z3MXt68QoRV35yGQr+4fB1l6Sd4Wt62Znj+zwBHCNpZjj674XA4ow0i9k3GvBC4EEzM0kTgbuBK81s75zu4VD0OyWdHvbK+jBBD7KSaixR76z0f4oPrd3G8k0d7OhOFT0GN3bt7OmlIWO2zoa6Wnb29JYoIlcuhi1EzGxe+DrezFrC1/TSksubmFkfcDlBz6o1wB1mtlrSNZLOD5PdDEyWtA64gn3zt18OHA1cJWlFuEwJj30CuAlYBzxPiXtmQdDFt7ff2NNXvB5a6QIk1TfApMYkqb4BL0jcqGppqKOnd//f6Z7efloa6koUkSsXceZYnwt8noyHDXO8sY6Z3UPQDTe676rI+m7gfVnOuxa4dpA8l1JmT803JdMj+fZTnyjOPOve1OAKbWZrE8s3dQDsd0/k2EMmlTgyV2pxbqzfBvwLsBIYKEw4lS/9vOODa7Zy1JRmZrY2FfzG486eXiZlvEdDXS0dkZrIju4UG7Z3sbOnl5aGuqLE5fJTjj+riY1JZk+fxIbtXXR0p2hpqOPYQ4p7U92VpziFyF/NLPM+hovY0Z1i684eIBiMMd2sNLvAPVjSTQ3pGgjs39SQbu5qTCaY1Jikp7e/KHG5+Mr5ZxUUJP774vYXpxD5N0k3AQ8QPCcCgJn9YtSjqlAbtncxoSH4I7t75Ssc3Jwk1T/Afc9s5dAJDaPzHi+kWNn/l/327e7t56XXe6irraGutobe/gF6+wc4bEIDD/9lOy+/3kPfgJGs3XcLbLTjGiy2clBJceX6s0omavjg6UfsncPGuVKJ8xv4D8BxQB37mrMM8EIktLOnlxmTG6lP1HDv6lcK90Z/eW74NKVSrrGNwbgStTVcOm/mKAbjXHxxCpFTzOzYgkUyBrQ01DGurpbbLztj777uVB/JRA0nT5s4Ku/x0EMP8ba3vS3WOSte3EGqb2C/5q7Rjivf2IqhkuLK9Wd13jf/yANrtnoh4kouTiHyZ0mzMgZNdBHZerDs6etn1mEtJGrjDFM2uNoaxc7r6CnNLN/UQW2NChZXvrEVQyXFlevP6pxZbfzgD+t5vaeXCd7N1pVQnL+s04EV4XDuT4dDsD9dqMAqUboHSzJRQ0d3imSipoxuiJZfXO5Auf6szjm+jb4B46Hn/lqiSJ0LxKmJ5DUMfLUp1x4s5RqXO1AuP6uTp02ktTnJ/c9s5fyTDitSZJWpHLtM5xJXucadKeeaiJltzLYUMjjnXHa1NeLtx01hydpt9Pb7Y1uDKdfRHIaLq1zjzqb8Goqdczk55/g2du3u44kNr5U6lLIVHc1B0t71DdtznpS1JHGVa9zZeCHiXIWad0wryUQN963ZWupQyla5Dhw5XFzlGnc2Xog4V6EakwnmHd3K/Wu2Ekzu6TKV68CRw8VVrnFn44WIcxXsnOPbePG1Hp7b2lnqUMrSzNYmulN9dKf6MLO96zNbm8o6rnKNOxsvRJyrYGcfH8yKcH8ZNGmV45w25dq9fbi4yjXubHzgHecqWFvLOE6aOoH712zlk2cdXbI4fODI+IaLq1zjzuQ1Eecq3DnHt7HixR1s27W7ZDFUUm8iN7q8EHGuwp0zqw0zWPLstpLFUEm9idzo8kLEuQp33CHjOXxiA/c9U7pCpJJ6E7nR5YWIcxVOEuccP4WH1/2VnlT/8CcUQCX1JnKjq2iFiKT54eCN6yRdmeV4vaTbw+OPSZoR7p8saYmkTknfzjinPcxzRbhMKc6nca68nDOrjd29A/xp3faSvH8l9SZyo6sovbMk1QI3AOcCm4EnJC3OGFb+UqDDzI6WdCFwHfB+YDfwReCEcMn0QTNbWtAP4FyZO23mZJrrE9y/ZivnzGorSQyV0pvIja5i1UROBdaZ2XozSwGLgAUZaRYAt4TrdwJnS5KZdZnZwwSFiXMui2SihrcdezD3r9nGwIA/ve6KR8UYLkHSQmC+mX0s3L4YOM3MLo+kWRWm2RxuPx+m2R5ufwSYm3FOOzAZ6Ad+DlxrWT6QpMuAywDa2trmLFq0aNiYOzs7aW5uzuvzFlK5xgXlG1u1xPXnl/q48ek9XHX6OI6cWDv8CUOolms2WqohrrPOOmuZmc094ICZFXwBFgI3RbYvBr6dkWYVMDWy/TzQGtn+SJZzDg9fxwO/Bz48XCxz5syxXCxZsiSndMVWrnGZlW9s1RJXR9ceO/Jzd9tXf/fsiPOqlms2WqohLmCpZflOLVZz1hZgWmR7argvaxpJCWAC8OpQmZrZlvB1F/BTgmYz56rSxMYkp8yYVBZDoLjqUaxC5AngGEkzJSWBC4HFGWkWA5eE6wuBB8PSLytJCUmt4Xod8LcEtRnnqtbfHN3Ks6/s4s6lL2Ydv6pU41uV47haY1n6enfu6Sv49S5KIWJmfcDlwL3AGuAOM1st6RpJ54fJbgYmS1oHXAHs7QYs6QXg68BHJG2WNAuoB+4N53lfQVCT+UExPo9z5WhHd4qDm4PeUWtf2VU2s+VV0ix9Y0H0etfWqODXu2gDMJrZPcA9GfuuiqzvBt43yLkzBsl2zmjF51yl27C9i5mtzUyb1ED7c39lcnM9qb5+1ry8i5mtTWzY3kVf/wDJxL6b7tHjUate6aNn5cujFleu7zulZRxvmj4RSaPy3tUoOo7ZawTzzqT3F6ILto/i69wYsbOnl0mNSd58dCu3P/Ei//67Z0eW4YonRyewmI6Z0swHT5vO382ZSss4HzYlrvTvQVRDXS0dlV4Tcc4VVnr8qotOmc5bjm7FDHb39VNXK447tIVnX95Jb78xLlIjiB6PeuKJpZxyyoG9OfMR532f3vw6tz26kat//QzX/W4tF8w+jA+edgRTJzWwYXvX3jb+ma1N/jT8INK/B+kaCBR2HDMvRJwbI2a2NrF8UwcA0w9qpKe3n+5U397hRw5pGbd3zo+GutoDjke9Mr6G4w5pyfY2scV53+MOaeHv505j5ebX+cmjG7lr+RZ+9viLHNnaxPwTDmFSt9G9dRcrN+/guENbGF+kmkpDXS1HTG6siGa26O8BsHccs2MPmVSQ9/NCxLkxIj1+1YbtXXR0p2hpqOPYQw6cLW+w46WKK5s3Tp3AdQtP5PPvOp5vPvAcv1v9Ct9pfz44+PjKgsY7mBMOb+FDpx3B+Scftt9/+eUmer37B4xkoqagP+fyvRLOudjKdba8fN93QmMdb/svB7PgpMNYu7WTdc+tYfLUozCDrj29vOHwCQWI9kAvv76bRY+/yJW/WMmX7lnDe980lQ+dPp3W5vqybGZLX+/X1yeYPb0wNZA0L0Scc2WtpaGO3X0DHH9oC+N31DL1qFa6U317Rwoulo+8eQZLN3Zw6yMbue2xjfzozy9w3CHjmf+GQzhiADp39/HY+lc5adrEUS1I6hM1Zd2M5oWIc66sFbuNfzCSOGXGQZwy4yC2d87iG/c/x72rt/KNB/4SJHjo8YK876xDW/j+xXOYdlBjQfIfKS9EnHNlrdht/Lloba7n3OPbeO/sqax4cQdPrXmOCVOmAkZPqp+ZB4/OoIe9/QPc/PAGzv/2w3zng3M446jJo5LvaPJCxDlX9orZxp+rloY6Un0DzJ1xEIf0JJh6/NSCNLO9+6TD+McfL+Ximx/j6vPfwIdOP2LU8h4NPj2uc87lITolMFCwKYFntjbxi0+8mbcc08oXfrmKL/xyJb39A6P6HiPhNRHnnMtDMZvZWsbVcdMlp/CVe5/l+w+tZ922Tr78d29kR3cvO3t6aWmoK1nPMK+JOOdcntIFSXN9ouBzytfWiM+ddzzXv/8knty4g7//3iOs29ZZ8kEtvSbinHMV5D2zp5LqG+Dff/ssX/zVKo6ZMh6A/gGjRtA8bt/X+o4dPZx8aqqghZvXRJxzrsIc0jKOr7/vJOYecRADZgyYIUFvvzFg7F2KMPu510Scc67SpHuGfXb+cXv3ZesZ1t7eXvD7JF4Tcc65ChPtGWZmBesZlgsvRJxzrsKkb+gnEzV0dKf21kBK0TvLm7Occ64ClWowzUxeE3HOOZe3ohUikuZLWitpnaQrsxyvl3R7ePwxSTPC/ZMlLZHUKenbGefMkbQyPOdbKuehLp1zbgwqSiEiqRa4ATgPmAVcJGlWRrJLgQ4zOxq4Hrgu3L8b+CLwmSxZfxf4R+CYcJk/+tE755wbTLFqIqcC68xsvZmlgEXAgow0C4BbwvU7gbMlycy6zOxhgsJkL0mHAi1m9qiZGfBj4IKCfgrnnHP7KVYhcjjwYmR7c7gvaxoz6wNeB4Ya9/jwMJ+h8nTOOVdAVdE7S9JlwGUAbW1ttLe3D3tOZ2dnTumKrVzjgvKNzeOKr1xj87jiKUZcxSpEtgDTIttTw33Z0myWlAAmAK8Ok+fUYfIEwMxuBG4EmDt3rp155pnDBtze3k4u6YqtXOOC8o3N44qvXGPzuOIpRlzFas56AjhG0kxJSeBCYHFGmsXAJeH6QuDB8F5HVmb2MrBT0ulhr6wPA78a/dCdc84NRkN8T4/uG0nvAr4B1AI/NLMvSboGWGpmiyWNA24FZgOvARea2frw3BeAFiAJ7ADeYWbPSJoL/AhoAH4L/NNQBU+Y11+BjTmE3Apsj/1BC69c44Lyjc3jiq9cY/O44hnNuI4ws4MzdxatEKk0kpaa2dxSx5GpXOOC8o3N44qvXGPzuOIpRlz+xLpzzrm8eSHinHMub16IDO7GUgcwiHKNC8o3No8rvnKNzeOKp+Bx+T0R55xzefOaiHPOubx5IeKccy5vXohkMdyw9aUi6YVw6PsVkpaWOJYfStomaVVk30GS7pP0l/B10lB5FDGuqyVtCa/bivCZpWLHNS2c0uAZSaslfTrcX9JrNkRcJb1mksZJelzSU2Fc/yvcPzOcKmJdOHVEUWdlGiKuH0naELleJxczrowYayUtl/SbcLuw18zMfIksBA9DPg8cSfBw41PArFLHFcb2AtBa6jjCWN4KvAlYFdn3FeDKcP1K4Loyietq4DMlvl6HAm8K18cDzxFMi1DSazZEXCW9ZoCA5nC9DngMOB24g+BBZIDvAR8vk7h+BCws5e9YJMYrgJ8Cvwm3C3rNvCZyoFyGra96ZvYHgpEFoqLD+d9CCYbmHySukjOzl83syXB9F7CGYNTpkl6zIeIqKQt0hpt14WLA2wmmioDSXK/B4ioLkqYC/xW4KdwWBb5mXogcKJdh60vFgN9LWhaOTFxu2iwY0wzgFaCtlMFkuFzS02FzV9Gb2aLCWTtnE/wXWzbXLCMuKPE1C5tlVgDbgPsIWgh2WDBVBJTobzMzLjNLX68vhdfrekn1xY4r9A3gfwID4fZkCnzNvBCpLPPM7E0EM0R+UtJbSx3QYCyoO5fLf2jfBY4CTgZeBr5WqkAkNQM/B/7ZzHZGj5XymmWJq+TXzMz6zexkghG6TwWOK3YM2WTGJekE4HME8Z0CHAR8tthxSfpbYJuZLSvm+3ohcqBchq0vCTPbEr5uA+4i+MMqJ1vDGSfTM09uK3E8AJjZ1vAPfwD4ASW6bpLqCL6obzOzX4S7S37NssVVLtcsjGUHsAQ4A5gYThUBJf7bjMQ1P2wWNDPbA/wHpblefwOcHw5Yu4igGeubFPiaeSFyoFyGrS86SU2SxqfXgXcAq4Y+q+iiw/lfQpkMzZ/+kg69hxJct7Bt+mZgjZl9PXKopNdssLhKfc0kHSxpYrjeAJxLcL9mCcFUEVCa65Utrmcj/wiI4J5D0X/HzOxzZjbVzGYQfG89aGYfpNDXrNQ9CcpxAd5F0EvleeBfSx1PGNORBD3FngJWlzou4GcEzRy9BO2slxK0vz4A/AW4HzioTOK6FVgJPE3wpX1oCeKaR9BU9TSwIlzeVeprNkRcJb1mwInA8vD9VwFXhfuPBB4H1gH/CdSXSVwPhtdrFfATwh5cpVqAM9nXO6ug18yHPXHOOZc3b85yzjmXNy9EnHPO5c0LEeecc3nzQsQ551zevBBxzjmXNy9EnHPO5c0LEeecc3nzQsRVFEkm6WuR7c9IunoU8p0RnYOkkCR9StIaSbdlOfZnSRMlfWKU3/OAPCX9eTTfw1UnL0RcpdkD/J2k1lIHEqVArn9PnwDOtWBIiv2Y2ZuBiWGa0YzhgDzD93JuRLwQcZWmD7gR+O/RnZk1iXQNJdz/bDjz3HOSbpN0jqQ/KZhNMDpQXiI8vkbSnZIaw7w+FM5mt0LS9yXVRt5zraQfEwx3MS0jpiskrQqXfw73fY9gGIrfStrvM4THO4F/B44K3++rcWOQ9MtwuoDVkSkDsuXZOVickbzXSPpBmNfvJTWE47jdrWB2v1WS3p/lc/xC0rWS/iBpk6Rzhvm5ukpVyvFdfPEl7gJ0Ai0EszxOAD5DMAvfDPafzTC6vw94I8E/TcuAHxLMULcA+GWYfgbBGFJ/E27/MMzjeODXQF24/zvAhyPnDACnZ4lzDsFYSk1AM8F4Z7PDYy8wyAyV4efL/CyxYiAcfwtoIChYJmfmGXmvoeJMX7uTw+07gA8B7wV+EMlnQpbP8RfCmREJBnD8j1L/7vhSmCU9PLBzFcPMdob/eX8K6MnhlA1mthJA0mrgATMzSSsJvijTXjSzP4XrPwnz303wRftEMEArDew/XPtGM3s0y3vOA+4ys67wfX8BvIVg8L64zo4Zw6ckvSdcnwYcQzDhVTbDxbnBzFaE68sIrtcdwNckXUcwyN8foxmGNbgJwPXhrjpgR86f1lUUL0RcpfoG8CTB3A0Q/MccbZ4dF1nfE1kfiGwPsP/fQOZopEZQY7nFzD43SBxdMWLOV84xSDoTOAc4w8y6JbWz/7WIK3rt+oEGM3tO0psIRvu9VtIDZnZNJN0sYJmZ9YfbJ1J+0xa4UeL3RFxFMrPXCP4jvjTctRWYImmygqlJ/zaPbKdLOiNc/wDwMMEw7QslTQGQdJCkI3LI64/ABZIaFcz/8p5wXy52AeMj23FimAB0hAXIccDpg+SZd5ySDgO6zewnwFeBN2UkeSPBkPJpJxIMne7GIC9EXCX7GtAKYGa9wDUE8ybcBzybR35rCaYdXgNMAr5rZs8AXyCY2/7pMO9Dh8iDMJ4ngR+F8TwG3GRmOTVlmdmrwJ/Cm9ZfjRnD7wg6CKwhuJn+aLY8RxjnG4HHFcwz/m/AtVmORwuRE/CayJjl84k455zLm9dEnHPO5c0LEeecc3nzQsQ551zevBBxzjmXNy9EnHPO5c0LEeecc3nzQsQ551ze/j+q56KJywbNdgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "result.plot_convergence();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "然后，我们想用`hiplot`绘制高维交互图。我们先整理出数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "data = result.plot_hi(return_data_only=True, target_name=\"accuracy\", loss2target_func=lambda loss: 1 - loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "如果直接绘图的话，您会发现由于算法选择的存在，会导致高维交互图非常的杂乱。我们简单编写一个函数，对不同的算法选择结果的超参进行可视化："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "import hiplot as hip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "def viz_subset(model_name, data):\n",
    "    data_filtered = []\n",
    "    for datum  in data:\n",
    "        if datum[\"classifier:__choice__\"] == model_name:\n",
    "            datum = datum.copy()\n",
    "            score = datum.pop(\"accuracy\")\n",
    "            AS, HP = layering_config(datum)[\"classifier\"].popitem()\n",
    "            HP[\"accuracy\"]  = score\n",
    "            data_filtered.append(HP)\n",
    "    hip.Experiment.from_iterable(data_filtered).display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "我们简单地对随机森林 `RandomForestClassifier` 进行可视化："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![hiplot](https://img-blog.csdnimg.cn/2020122709574745.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因为绘制`hiplot`高维交互图后`jupyter notebook`的文件大小会大量增加，所以我们以截图代替。您可以在自己的`notebook`中取消注释并执行以下代码："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "#viz_subset(\"RandomForestClassifier\", data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "#viz_subset(\"LinearSVC\", data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "#viz_subset(\"KNeighborsClassifier\", data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "我们将所有代码整理为了 `05. Implement a Simple AutoML System.py` 脚本。\n",
    "\n",
    "您可以在这个脚本中更直接地学习一个简单AutoML系统的搭建方法。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "auto-sklearn",
   "language": "python",
   "name": "auto-sklearn"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}