practice_solution.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#XVFB will be launched if you run on a server\n",
    "import os\n",
    "if type(os.environ.get(\"DISPLAY\")) is not str or len(os.environ.get(\"DISPLAY\"))==0:\n",
    "    !bash ./xvfb start\n",
    "    %env DISPLAY=:1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reinforcement Learning 101\n",
    "\n",
    "![img](https://s1.postimg.org/4ndvvz3o33/0a13e53e-0d6b-49bf-9362-e9826cbf95d8.png)\n",
    "_(image from berkeley cs188x)_\n",
    "\n",
    "This notebook will teach you to solve reinforcement learning problems with crossentropy method. We'll follow-up by scaling everything up and using neural network policy.\n",
    "\n",
    "Both experiments will take place in [openAI Gym](https://gym.openai.com/) environments (install: `pip install gym` )."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+\n",
      "|R: | : :\u001b[35mG\u001b[0m|\n",
      "| : : : : |\n",
      "|\u001b[43m \u001b[0m: : : : |\n",
      "| | : | : |\n",
      "|\u001b[34;1mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "import numpy as np, pandas as pd\n",
    "\n",
    "#create a single game instance\n",
    "env = gym.make(\"Taxi-v2\")\n",
    "\n",
    "#start new game\n",
    "s0 = env.reset()\n",
    "\n",
    "# display the game state\n",
    "env.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gym interface\n",
    "\n",
    "![img](https://cdn-images-1.medium.com/max/1600/1*HvoLc50Dpq1ESKuejhICHg.png)\n",
    "\n",
    "The three main methods of an environment are\n",
    "* __reset()__ - reset environment to initial state, _return first observation_\n",
    "* __render()__ - show current environment state (a more colorful version :) )\n",
    "* __step(a)__ - commit action __a__ and return (new observation, reward, is done, info)\n",
    " * _new observation_ - an observation right after commiting the action __a__\n",
    " * _reward_ - a number representing your reward for commiting action __a__\n",
    " * _is done_ - True if the MDP has just finished, False if still in progress\n",
    " * _info_ - some auxilary stuff about what just happened. Ignore it for now"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n_states=500, n_actions=6\n"
     ]
    }
   ],
   "source": [
    "n_states = env.observation_space.n\n",
    "n_actions = env.action_space.n\n",
    "\n",
    "print(\"n_states=%i, n_actions=%i\"%(n_states,n_actions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "new observation code: 483\n",
      "reward: -1\n",
      "is game over?: False\n",
      "printing new state:\n",
      "+---------+\n",
      "|\u001b[34;1mR\u001b[0m: | : :G|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|Y| : |\u001b[35mB\u001b[0m:\u001b[43m \u001b[0m|\n",
      "+---------+\n",
      "  (South)\n"
     ]
    }
   ],
   "source": [
    "actions = ['South','North','East','West','Pickup','Dropoff']\n",
    "\n",
    "# Here we pick action #1: north:\n",
    "new_obs, reward, is_done, _ = env.step(actions.index(\"South\"))\n",
    "\n",
    "print(\"new observation code:\", new_obs)\n",
    "print(\"reward:\", reward)\n",
    "print(\"is game over?:\", is_done)\n",
    "\n",
    "print(\"printing new state:\")\n",
    "env.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create stochastic policy\n",
    "\n",
    "This time our policy should be a probability distribution.\n",
    "\n",
    "```policy[s,a] = P(take action a | in state s)```\n",
    "\n",
    "Since we still use integer state and action representations, you can use a 2-dimensional array to represent the policy.\n",
    "\n",
    "Please initialize policy __uniformly__, that is, probabililities of all actions should be equal.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'n_states' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-3-a5d47870ab27>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mpolicy\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mones\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mn_states\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mn_actions\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0mn_actions\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'n_states' is not defined"
     ]
    }
   ],
   "source": [
    "policy = np.ones([n_states,n_actions])/n_actions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "assert type(policy) in (np.ndarray,np.matrix)\n",
    "assert np.allclose(policy,1./n_actions)\n",
    "assert np.allclose(np.sum(policy,axis=1), 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Play the game\n",
    "\n",
    "Just like before, but we also record all states and actions we took."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generate_session(policy,t_max=10**4):\n",
    "    \"\"\"\n",
    "    Play game until end or for t_max ticks.\n",
    "    :param policy: an array of shape [n_states,n_actions] with action probabilities\n",
    "    :returns: list of states, list of actions and sum of rewards\n",
    "    \"\"\"\n",
    "    states,actions = [],[]\n",
    "    total_reward = 0.\n",
    "    \n",
    "    s = env.reset()\n",
    "    \n",
    "    for t in range(t_max):\n",
    "        \n",
    "        a = np.random.choice(range(n_actions),p=policy[s0])\n",
    "        \n",
    "        new_s,r,done,info = env.step(a)\n",
    "        \n",
    "        #Record state, action and add up reward to states,actions and total_reward accordingly. \n",
    "        states.append(s)\n",
    "        actions.append(a)\n",
    "        total_reward+=r\n",
    "        \n",
    "        s = new_s\n",
    "        if done:\n",
    "            break\n",
    "    return states,actions,total_reward\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'policy' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-5-1a02f45ccfc4>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgenerate_session\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpolicy\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;32massert\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;32massert\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;32massert\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mr\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32min\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mfloat\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfloat\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'policy' is not defined"
     ]
    }
   ],
   "source": [
    "s,a,r = generate_session(policy)\n",
    "assert type(s) == type(a) == list\n",
    "assert len(s) == len(a)\n",
    "assert type(r) in [float,np.float]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'policy' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-6-3adc04105705>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmagic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'matplotlib inline'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0msample_rewards\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mgenerate_session\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpolicy\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mt_max\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1000\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0m_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m200\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msample_rewards\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mbins\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m20\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m;\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-6-3adc04105705>\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmagic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'matplotlib inline'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0msample_rewards\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mgenerate_session\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpolicy\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mt_max\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1000\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0m_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m200\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msample_rewards\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mbins\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m20\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m;\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'policy' is not defined"
     ]
    }
   ],
   "source": [
    "#let's see the initial reward distribution\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "sample_rewards = [generate_session(policy,t_max=1000)[-1] for _ in range(200)]\n",
    "\n",
    "plt.hist(sample_rewards,bins=20);\n",
    "plt.vlines([np.percentile(sample_rewards,50)],[0],[100],label=\"50'th percentile\",color='green')\n",
    "plt.vlines([np.percentile(sample_rewards,90)],[0],[100],label=\"90'th percentile\",color='red')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Crossentropy method steps (2pts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def select_elites(states_batch,actions_batch,rewards_batch,percentile=50):\n",
    "    \"\"\"\n",
    "    Select states and actions from games that have rewards >= percentile\n",
    "    :param states_batch: list of lists of states, states_batch[session_i][t]\n",
    "    :param actions_batch: list of lists of actions, actions_batch[session_i][t]\n",
    "    :param rewards_batch: list of rewards, rewards_batch[session_i][t]\n",
    "    \n",
    "    :returns: elite_states,elite_actions, both 1D lists of states and respective actions from elite sessions\n",
    "    \n",
    "    Please return elite states and actions in their original order \n",
    "    [i.e. sorted by session number and timestep within session]\n",
    "    \n",
    "    If you're confused, see examples below. Please don't assume that states are integers (they'll get different later).\n",
    "    \"\"\"\n",
    "    \n",
    "    reward_threshold = np.percentile(rewards_batch,percentile)\n",
    "    \n",
    "    elite_states = []\n",
    "    elite_actions = []\n",
    "    for states,actions,R in zip(states_batch,actions_batch,rewards_batch):\n",
    "        if R >= reward_threshold:\n",
    "            elite_states.extend(states)\n",
    "            elite_actions.extend(actions)\n",
    "    \n",
    "    return elite_states,elite_actions\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ok!\n"
     ]
    }
   ],
   "source": [
    "states_batch = [\n",
    "    [1,2,3],   #game1\n",
    "    [4,2,0,2], #game2\n",
    "    [3,1]      #game3\n",
    "]\n",
    "\n",
    "actions_batch = [\n",
    "    [0,2,4],   #game1\n",
    "    [3,2,0,1], #game2\n",
    "    [3,3]      #game3\n",
    "]\n",
    "rewards_batch = [\n",
    "    3,         #game1\n",
    "    4,         #game2\n",
    "    5,         #game3\n",
    "]\n",
    "\n",
    "test_result_0 = select_elites(states_batch,actions_batch,rewards_batch,percentile=0)\n",
    "test_result_40 = select_elites(states_batch,actions_batch,rewards_batch,percentile=30)\n",
    "test_result_90 = select_elites(states_batch,actions_batch,rewards_batch,percentile=90)\n",
    "test_result_100 = select_elites(states_batch,actions_batch,rewards_batch,percentile=100)\n",
    "\n",
    "assert np.all(test_result_0[0] == [1, 2, 3, 4, 2, 0, 2, 3, 1])  \\\n",
    "   and np.all(test_result_0[1] == [0, 2, 4, 3, 2, 0, 1, 3, 3]),\\\n",
    "        \"For percentile 0 you should return all states and actions in chronological order\"\n",
    "assert np.all(test_result_40[0] == [4, 2, 0, 2, 3, 1]) and \\\n",
    "        np.all(test_result_40[1] ==[3, 2, 0, 1, 3, 3]),\\\n",
    "        \"For percentile 30 you should only select states/actions from two first\"\n",
    "assert np.all(test_result_90[0] == [3,1]) and \\\n",
    "        np.all(test_result_90[1] == [3,3]),\\\n",
    "        \"For percentile 90 you should only select states/actions from one game\"\n",
    "assert np.all(test_result_100[0] == [3,1]) and\\\n",
    "       np.all(test_result_100[1] == [3,3]),\\\n",
    "        \"Please make sure you use >=, not >. Also double-check how you compute percentile.\"\n",
    "print(\"Ok!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def update_policy(elite_states,elite_actions):\n",
    "    \"\"\"\n",
    "    Given old policy and a list of elite states/actions from select_elites,\n",
    "    return new updated policy where each action probability is proportional to\n",
    "    \n",
    "    policy[s_i,a_i] ~ #[occurences of si and ai in elite states/actions]\n",
    "    \n",
    "    Don't forget to normalize policy to get valid probabilities and handle 0/0 case.\n",
    "    In case you never visited a state, set probabilities for all actions to 1./n_actions\n",
    "    \n",
    "    :param elite_states: 1D list of states from elite sessions\n",
    "    :param elite_actions: 1D list of actions from elite sessions\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    new_policy = np.zeros([n_states,n_actions])\n",
    "    \n",
    "    for state,action in zip(elite_states,elite_actions):\n",
    "        new_policy[state,action]+=1\n",
    "    \n",
    "    row_totals = new_policy.sum(axis=1,keepdims=True)\n",
    "    new_policy = new_policy/row_totals\n",
    "    \n",
    "    new_policy[np.isnan(new_policy)] = 1/n_actions\n",
    "    return new_policy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'n_states' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-10-d881b43947c7>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mnew_policy\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mupdate_policy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0melite_states\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0melite_actions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;32massert\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0misfinite\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnew_policy\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mall\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"Your new policy contains NaNs or +-inf. Make sure you don't divide by zero.\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-9-9d0b8c2cbc7c>\u001b[0m in \u001b[0;36mupdate_policy\u001b[0;34m(elite_states, elite_actions)\u001b[0m\n\u001b[1;32m     14\u001b[0m     \"\"\"\n\u001b[1;32m     15\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 16\u001b[0;31m     \u001b[0mnew_policy\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mn_states\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mn_actions\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     17\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     18\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mstate\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0maction\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0melite_states\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0melite_actions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'n_states' is not defined"
     ]
    }
   ],
   "source": [
    "\n",
    "elite_states, elite_actions = ([1, 2, 3, 4, 2, 0, 2, 3, 1], [0, 2, 4, 3, 2, 0, 1, 3, 3])\n",
    "\n",
    "\n",
    "new_policy = update_policy(elite_states,elite_actions)\n",
    "\n",
    "assert np.isfinite(new_policy).all(), \"Your new policy contains NaNs or +-inf. Make sure you don't divide by zero.\"\n",
    "assert np.all(new_policy>=0), \"Your new policy can't have negative action probabilities\"\n",
    "assert np.allclose(new_policy.sum(axis=-1),1), \"Your new policy should be a valid probability distribution over actions\"\n",
    "reference_answer = np.array([\n",
    "       [ 1.        ,  0.        ,  0.        ,  0.        ,  0.        ],\n",
    "       [ 0.5       ,  0.        ,  0.        ,  0.5       ,  0.        ],\n",
    "       [ 0.        ,  0.33333333,  0.66666667,  0.        ,  0.        ],\n",
    "       [ 0.        ,  0.        ,  0.        ,  0.5       ,  0.5       ]])\n",
    "assert np.allclose(new_policy[:4,:5],reference_answer)\n",
    "print(\"Ok!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training loop\n",
    "Generate sessions, select N best and fit to those."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import clear_output\n",
    "\n",
    "def show_progress(batch_rewards,log, reward_range=[-990,+10]):\n",
    "    \"\"\"\n",
    "    A convenience function that displays training progress. \n",
    "    No cool math here, just charts.\n",
    "    \"\"\"\n",
    "    \n",
    "    mean_reward,threshold = np.mean(batch_rewards),np.percentile(batch_rewards,percentile)\n",
    "    log.append([mean_reward,threshold])\n",
    "\n",
    "    clear_output(True)\n",
    "    print(\"mean reward = %.3f, threshold=%.3f\"%(mean_reward,threshold))\n",
    "    plt.figure(figsize=[8,4])\n",
    "    plt.subplot(1,2,1)\n",
    "    plt.plot(list(zip(*log))[0],label='Mean rewards')\n",
    "    plt.plot(list(zip(*log))[1],label='Reward thresholds')\n",
    "    plt.legend()\n",
    "    plt.grid()\n",
    "    \n",
    "    plt.subplot(1,2,2)\n",
    "    plt.hist(batch_rewards,range=reward_range);\n",
    "    plt.vlines([np.percentile(batch_rewards,percentile)],[0],[100],label=\"percentile\",color='red')\n",
    "    plt.legend()\n",
    "    plt.grid()\n",
    "\n",
    "    plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'n_states' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-12-ab3a22b38888>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m#reset policy just in case\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mpolicy\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mones\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mn_states\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mn_actions\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0mn_actions\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'n_states' is not defined"
     ]
    }
   ],
   "source": [
    "#reset policy just in case\n",
    "policy = np.ones([n_states,n_actions])/n_actions "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'policy' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<timed exec>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32m<timed exec>\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'policy' is not defined"
     ]
    },
    {
     "ename": "NameError",
     "evalue": "name 'sessions' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-13-e3ac16d59fa1>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      9\u001b[0m     \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmagic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'time sessions = [generate_session(policy) for _ in range(n_sessions)]'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m     \u001b[0mbatch_states\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mbatch_actions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mbatch_rewards\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0msessions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     12\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m     \u001b[0melite_states\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0melite_actions\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mselect_elites\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch_states\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mbatch_actions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mbatch_rewards\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'sessions' is not defined"
     ]
    }
   ],
   "source": [
    "n_sessions = 250  #sample this many sessions\n",
    "percentile = 50  #take this percent of session with highest rewards\n",
    "learning_rate = 0.5  #add this thing to all counts for stability\n",
    "\n",
    "log = []\n",
    "\n",
    "for i in range(100):\n",
    "    \n",
    "    %time sessions = [generate_session(policy) for _ in range(n_sessions)]\n",
    "    \n",
    "    batch_states,batch_actions,batch_rewards = zip(*sessions)\n",
    "\n",
    "    elite_states, elite_actions = select_elites(batch_states,batch_actions,batch_rewards)\n",
    "    \n",
    "    new_policy = update_policy(elite_states, elite_actions)\n",
    "    \n",
    "    policy = learning_rate*new_policy + (1-learning_rate)*policy\n",
    "    \n",
    "    #display results on chart\n",
    "    show_progress(batch_rewards,log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Digging deeper: approximate crossentropy with neural nets\n",
    "\n",
    "![img](https://casd35.wikispaces.com/file/view/digging_deeper_final.jpg/359658499/503x260/digging_deeper_final.jpg)\n",
    "\n",
    "In this section we will train a neural network policy for continuous state space game"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f57a97d53c8>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD8CAYAAAB9y7/cAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEohJREFUeJzt3X+s3fV93/Hna5hAlmQ1hAvybDOT1ltDp8XQO+KIaaKQ\ntsCqmkrNBJsaFCFdJhEpUaOt0ElrIg2pldawRdtQ3ELjTFkII0mxEGvqOURV/gjkkjiOjUNxEiu+\ntYdvFiDJorGZvPfH+dxwao7vPb4/fH0/eT6ko/P9fr6f8z3vDz687tef+/34pKqQJPXnb6x2AZKk\nlWHAS1KnDHhJ6pQBL0mdMuAlqVMGvCR1asUCPsmNSZ5NcjjJ3Sv1PpKk0bIS98EnOQ/4S+CXgRng\ny8BtVfXMsr+ZJGmklbqCvwY4XFXfqqr/CzwE7Fih95IkjbBuhc67ETg6tD8DvP10nS+55JLasmXL\nCpUiSWvPkSNH+O53v5ulnGOlAn5UUX9tLijJFDAFcPnllzM9Pb1CpUjS2jM5Obnkc6zUFM0MsHlo\nfxNwbLhDVe2sqsmqmpyYmFihMiTpp9dKBfyXga1JrkjyOuBWYPcKvZckaYQVmaKpqpNJ3gt8DjgP\neLCqDq7Ee0mSRlupOXiq6nHg8ZU6vyRpfq5klaROGfCS1CkDXpI6ZcBLUqcMeEnqlAEvSZ0y4CWp\nUwa8JHXKgJekThnwktQpA16SOmXAS1KnDHhJ6pQBL0mdMuAlqVMGvCR1yoCXpE4Z8JLUqSV9ZV+S\nI8APgFeAk1U1meRi4FPAFuAI8E+r6oWllSlJOlPLcQX/S1W1raom2/7dwN6q2grsbfuSpLNsJaZo\ndgC72vYu4JYVeA9J0gKWGvAF/HmSp5NMtbbLquo4QHu+dInvIUlahCXNwQPXVtWxJJcCe5J8Y9wX\nth8IUwCXX375EsuQJJ1qSVfwVXWsPZ8APgtcAzyfZANAez5xmtfurKrJqpqcmJhYShmSpBEWHfBJ\n3pDkTXPbwK8AB4DdwO2t2+3Ao0stUpJ05pYyRXMZ8Nkkc+f5r1X1Z0m+DDyc5A7gO8C7ll6mJOlM\nLTrgq+pbwNtGtP8v4IalFCVJWjpXskpSpwx4SeqUAS9JnTLgJalTBrwkdcqAl6ROGfCS1CkDXpI6\nZcBLUqcMeEnqlAEvSZ0y4CWpUwa8JHXKgJekThnwktQpA16SOmXAS1KnDHhJ6pQBL0mdWjDgkzyY\n5ESSA0NtFyfZk+S59nxRa0+SjyQ5nGR/kqtXsnhJ0umNcwX/MeDGU9ruBvZW1VZgb9sHuAnY2h5T\nwP3LU6Yk6UwtGPBV9RfA905p3gHsatu7gFuG2j9eA18C1ifZsFzFSpLGt9g5+Muq6jhAe760tW8E\njg71m2ltr5FkKsl0kunZ2dlFliFJOp3l/iVrRrTVqI5VtbOqJqtqcmJiYpnLkCQtNuCfn5t6ac8n\nWvsMsHmo3ybg2OLLkyQt1mIDfjdwe9u+HXh0qP3d7W6a7cBLc1M5kqSza91CHZJ8ErgOuCTJDPB7\nwO8DDye5A/gO8K7W/XHgZuAw8CPgPStQsyRpDAsGfFXddppDN4zoW8BdSy1KkrR0rmSVpE4Z8JLU\nKQNekjplwEtSpwx4SeqUAS9JnTLgJalTBrwkdcqAl6ROGfCS1CkDXpI6ZcBLUqcMeEnqlAEvSZ0y\n4CWpUwa8JHXKgJekThnwktSpBQM+yYNJTiQ5MNT2wSR/lWRfe9w8dOyeJIeTPJvkV1eqcEnS/Ma5\ngv8YcOOI9vuqalt7PA6Q5ErgVuAX2mv+c5LzlqtYSdL4Fgz4qvoL4Htjnm8H8FBVvVxV3wYOA9cs\noT5J0iItZQ7+vUn2tymci1rbRuDoUJ+Z1vYaSaaSTCeZnp2dXUIZkqRRFhvw9wM/C2wDjgN/2Noz\nom+NOkFV7ayqyaqanJiYWGQZkqTTWVTAV9XzVfVKVf0Y+CNenYaZATYPdd0EHFtaiZKkxVhUwCfZ\nMLT7G8DcHTa7gVuTXJDkCmAr8NTSSpQkLca6hTok+SRwHXBJkhng94DrkmxjMP1yBLgToKoOJnkY\neAY4CdxVVa+sTOmSpPksGPBVdduI5gfm6X8vcO9SipIkLZ0rWSWpUwa8JHXKgJekThnwktQpA16S\nOmXAS1KnFrxNUvpp8/TOO1/T9otTH12FSqSl8QpeGjIq3KW1yoCXpE4Z8JLUKQNekjplwEtSpwx4\nSeqUAS9JnTLgJalTBrwkdcqAl6ROGfCS1KkFAz7J5iRPJDmU5GCS97X2i5PsSfJce76otSfJR5Ic\nTrI/ydUrPQhJ0muNcwV/EvhAVb0V2A7cleRK4G5gb1VtBfa2fYCbgK3tMQXcv+xVS2eR/9CY1qoF\nA76qjlfVV9r2D4BDwEZgB7CrddsF3NK2dwAfr4EvAeuTbFj2yiVJ8zqjOfgkW4CrgCeBy6rqOAx+\nCACXtm4bgaNDL5tpbaeeayrJdJLp2dnZM69ckjSvsQM+yRuBTwPvr6rvz9d1RFu9pqFqZ1VNVtXk\nxMTEuGVIksY0VsAnOZ9BuH+iqj7Tmp+fm3ppzyda+wyweejlm4Bjy1OuJGlc49xFE+AB4FBVfXjo\n0G7g9rZ9O/DoUPu7290024GX5qZyJElnzzhf2Xct8FvA15Psa22/C/w+8HCSO4DvAO9qxx4HbgYO\nAz8C3rOsFUuSxrJgwFfVFxk9rw5ww4j+Bdy1xLokSUvkSlZJ6pQBL0mdMuAlqVMGvCR1yoCXmqd3\n3rnaJUjLyoCXpE4Z8JLUKQNekjplwEtSpwx4SeqUAS9JnTLgJalTBrwkdcqAl6ROGfCS1CkDXpI6\nZcBL8/jFqY+udgnSohnwktSpcb50e3OSJ5IcSnIwyfta+weT/FWSfe1x89Br7klyOMmzSX51JQcg\nSRptnC/dPgl8oKq+kuRNwNNJ9rRj91XVvxvunORK4FbgF4C/DfyPJH+3ql5ZzsIlSfNb8Aq+qo5X\n1Vfa9g+AQ8DGeV6yA3ioql6uqm8Dh4FrlqNYSdL4zmgOPskW4Crgydb03iT7kzyY5KLWthE4OvSy\nGeb/gSBJWgFjB3ySNwKfBt5fVd8H7gd+FtgGHAf+cK7riJfXiPNNJZlOMj07O3vGhUuS5jdWwCc5\nn0G4f6KqPgNQVc9X1StV9WPgj3h1GmYG2Dz08k3AsVPPWVU7q2qyqiYnJiaWMgZJ0gjj3EUT4AHg\nUFV9eKh9w1C33wAOtO3dwK1JLkhyBbAVeGr5SpYkjWOcu2iuBX4L+HqSfa3td4HbkmxjMP1yBLgT\noKoOJnkYeIbBHTh3eQeNJJ19CwZ8VX2R0fPqj8/zmnuBe5dQlyRpiVzJKkmdMuAl4Omdd652CdKy\nM+AlqVMGvCR1yoCXpE4Z8JLUKQNekjplwEtSpwx4SeqUAS9JnTLgJalTBrwkdcqAl6ROGfDqVpKx\nH6NM3rlzSa+XVpsBL0mdGucLP6SfCo8dn/rJ9q9t2LmKlUjLwyt4ib8e7qP2pbXIgJekTo3zpdsX\nJnkqydeSHEzyodZ+RZInkzyX5FNJXtfaL2j7h9vxLSs7BEnSKONcwb8MXF9VbwO2ATcm2Q78AXBf\nVW0FXgDuaP3vAF6oqp8D7mv9pHPaqXPuzsGrB+N86XYBP2y757dHAdcD/6y17wI+CNwP7GjbAI8A\n/zFJ2nmkc9LknTuBV0P9g6tWibR8xpqDT3Jekn3ACWAP8E3gxao62brMABvb9kbgKEA7/hLw5uUs\nWpK0sLECvqpeqaptwCbgGuCto7q151GrPl5z9Z5kKsl0kunZ2dlx65UkjemM7qKpqheBLwDbgfVJ\n5qZ4NgHH2vYMsBmgHf8Z4HsjzrWzqiaranJiYmJx1UuSTmucu2gmkqxv268H3gkcAp4AfrN1ux14\ntG3vbvu04593/l2Szr5xVrJuAHYlOY/BD4SHq+qxJM8ADyX5t8BXgQda/weA/5LkMIMr91tXoG5J\n0gLGuYtmP3DViPZvMZiPP7X9/wDvWpbqJEmL5kpWSeqUAS9JnTLgJalT/nPB6pY3b+mnnVfwktQp\nA16SOmXAS1KnDHhJ6pQBL0mdMuAlqVMGvCR1yoCXpE4Z8JLUKQNekjplwEtSpwx4SeqUAS9JnTLg\nJalT43zp9oVJnkrytSQHk3yotX8sybeT7GuPba09ST6S5HCS/UmuXulBSJJea5x/D/5l4Pqq+mGS\n84EvJvnv7di/rKpHTul/E7C1Pd4O3N+eJUln0YJX8DXww7Z7fnvM900KO4CPt9d9CVifZMPSS5Uk\nnYmx5uCTnJdkH3AC2FNVT7ZD97ZpmPuSXNDaNgJHh14+09okSWfRWAFfVa9U1TZgE3BNkr8P3AP8\nPPAPgYuB32ndM+oUpzYkmUoynWR6dnZ2UcVLkk7vjO6iqaoXgS8AN1bV8TYN8zLwJ8A1rdsMsHno\nZZuAYyPOtbOqJqtqcmJiYlHFS5JOb5y7aCaSrG/brwfeCXxjbl49SYBbgAPtJbuBd7e7abYDL1XV\n8RWpXpJ0WuPcRbMB2JXkPAY/EB6uqseSfD7JBIMpmX3Av2j9HwduBg4DPwLes/xlS5IWsmDAV9V+\n4KoR7defpn8Bdy29NEnSUriSVZI6ZcBLUqcMeEnqlAEvSZ0y4CWpUwa8JHXKgJekThnwktQpA16S\nOmXAS1KnDHhJ6pQBL0mdMuAlqVMGvCR1yoCXpE4Z8JLUKQNekjplwEtSpwx4SerU2AGf5LwkX03y\nWNu/IsmTSZ5L8qkkr2vtF7T9w+34lpUpXZI0nzO5gn8fcGho/w+A+6pqK/ACcEdrvwN4oap+Driv\n9ZMknWVjBXySTcA/Af647Qe4HnikddkF3NK2d7R92vEbWn9J0lm0bsx+/x74V8Cb2v6bgRer6mTb\nnwE2tu2NwFGAqjqZ5KXW/7vDJ0wyBUy13ZeTHFjUCM59l3DK2DvR67ig37E5rrXl7ySZqqqdiz3B\nggGf5NeAE1X1dJLr5ppHdK0xjr3aMCh6Z3uP6aqaHKviNabXsfU6Luh3bI5r7UkyTcvJxRjnCv5a\n4NeT3AxcCPwtBlf065Osa1fxm4Bjrf8MsBmYSbIO+Bnge4stUJK0OAvOwVfVPVW1qaq2ALcCn6+q\nfw48Afxm63Y78Gjb3t32acc/X1WvuYKXJK2spdwH/zvAbyc5zGCO/YHW/gDw5tb+28DdY5xr0X8F\nWQN6HVuv44J+x+a41p4ljS1eXEtSn1zJKkmdWvWAT3JjkmfbytdxpnPOKUkeTHJi+DbPJBcn2dNW\n+e5JclFrT5KPtLHuT3L16lU+vySbkzyR5FCSg0ne19rX9NiSXJjkqSRfa+P6UGvvYmV2ryvOkxxJ\n8vUk+9qdJWv+swiQZH2SR5J8o/2/9o7lHNeqBnyS84D/BNwEXAncluTK1axpET4G3HhK293A3rbK\ndy+v/h7iJmBre0wB95+lGhfjJPCBqnorsB24q/3ZrPWxvQxcX1VvA7YBNybZTj8rs3tecf5LVbVt\n6JbItf5ZBPgPwJ9V1c8Db2PwZ7d846qqVXsA7wA+N7R/D3DPata0yHFsAQ4M7T8LbGjbG4Bn2/ZH\ngdtG9TvXHwzukvrlnsYG/E3gK8DbGSyUWdfaf/K5BD4HvKNtr2v9stq1n2Y8m1ogXA88xmBNypof\nV6vxCHDJKW1r+rPI4Jbzb5/63305x7XaUzQ/WfXaDK+IXcsuq6rjAO350ta+Jsfb/vp+FfAkHYyt\nTWPsA04Ae4BvMubKbGBuZfa5aG7F+Y/b/tgrzjm3xwWDxZJ/nuTptgoe1v5n8S3ALPAnbVrtj5O8\ngWUc12oH/FirXjuy5sab5I3Ap4H3V9X35+s6ou2cHFtVvVJV2xhc8V4DvHVUt/a8JsaVoRXnw80j\nuq6pcQ25tqquZjBNcVeSfzxP37UytnXA1cD9VXUV8L+Z/7byMx7Xagf83KrXOcMrYtey55NsAGjP\nJ1r7mhpvkvMZhPsnquozrbmLsQFU1YvAFxj8jmF9W3kNo1dmc46vzJ5bcX4EeIjBNM1PVpy3Pmtx\nXABU1bH2fAL4LIMfzGv9szgDzFTVk23/EQaBv2zjWu2A/zKwtf2m/3UMVsruXuWalsPwat5TV/m+\nu/02fDvw0txfxc41ScJg0dqhqvrw0KE1PbYkE0nWt+3XA+9k8IutNb0yuzpecZ7kDUneNLcN/Apw\ngDX+Wayq/wkcTfL3WtMNwDMs57jOgV803Az8JYN50H+92vUsov5PAseB/8fgJ+wdDOYy9wLPteeL\nW98wuGvom8DXgcnVrn+ecf0jBn/92w/sa4+b1/rYgH8AfLWN6wDwb1r7W4CngMPAfwMuaO0Xtv3D\n7fhbVnsMY4zxOuCxXsbVxvC19jg4lxNr/bPYat0GTLfP458CFy3nuFzJKkmdWu0pGknSCjHgJalT\nBrwkdcqAl6ROGfCS1CkDXpI6ZcBLUqcMeEnq1P8HvvOFSz1CKGcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f57b314fcc0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "env = gym.make(\"CartPole-v0\").env  #if you see \"<classname> has no attribute .env\", remove .env or update gym\n",
    "\n",
    "env.reset()\n",
    "n_actions = env.action_space.n\n",
    "\n",
    "plt.imshow(env.render(\"rgb_array\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jheuristic/anaconda3/lib/python3.6/site-packages/sklearn/neural_network/multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (1) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n"
     ]
    }
   ],
   "source": [
    "#create agent\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "agent = MLPClassifier(hidden_layer_sizes=(20,20),\n",
    "                      activation='tanh',\n",
    "                      warm_start=True, #keep progress between .fit(...) calls\n",
    "                      max_iter=1 #make only 1 iteration on each .fit(...)\n",
    "                     )\n",
    "#initialize agent to the dimension of state an amount of actions\n",
    "agent.fit([env.reset()]*n_actions,range(n_actions));\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.46910669,  0.53089331])"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " agent.predict_proba([env.reset()])[0] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generate_session(env,t_max=1000):\n",
    "    \n",
    "    states,actions = [],[]\n",
    "    total_reward = 0\n",
    "    \n",
    "    s = env.reset()\n",
    "    \n",
    "    for t in range(t_max):\n",
    "        \n",
    "        #predict array of action probabilities\n",
    "        probs = agent.predict_proba([s])[0] \n",
    "        \n",
    "        a = np.random.choice([0,1],p=probs)\n",
    "        \n",
    "        new_s,r,done,info = env.step(a)\n",
    "        \n",
    "        #record sessions like you did before\n",
    "        states.append(s)\n",
    "        actions.append(a)\n",
    "        total_reward+=r\n",
    "        \n",
    "        s = new_s\n",
    "        #if done: break\n",
    "    return states,actions,total_reward\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean reward = 987.790, threshold=1000.000\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAroAAAFkCAYAAADPIZjbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzs3Xd4VFX6wPHvGRIISYAESei9hCZogihIEEQREUaWVZqK\ngCtYqPoDKxjBgugCS1UXBBugIosVgRWkiApOwLKGiIAgPaEESOg5vz8mGTKZhCSTaffO+3meeSDn\nnrnzvpebk5ebc89VWmuEEEIIIYQwG4u/AxBCCCGEEMIbpNAVQgghhBCmJIWuEEIIIYQwJSl0hRBC\nCCGEKUmhK4QQQgghTEkKXSGEEEIIYUpS6AohhBBCCFOSQlcIIYQQQpiSFLpCCCGEEMKUpNAVQggh\nhBCmVOJCVymVqJT6VCm1XymVrZSyFtBnolLqgFIqSym1WinVKN/2ckqp2UqpdKXUKaXUUqVUbL4+\n0Uqp95VSGUqp40qpeUqpiJKnKIQQxuOrsVYIIczMnSu6EcA24BFA59+olHoCGA4MBdoCmcBKpVTZ\nPN2mA3cAfwc6AjWAj/PtahHQDOiS07cj8IYb8QohhBH5aqwVQgjTUlq7jJ/Ff7NS2UAvrfWnedoO\nAK9qraflfF0ROAzcr7X+MOfrNKCf1vo/OX3igBTgBq31ZqVUM+B/QILWemtOn9uAL4BaWutDbgct\nhBAG462x1td5CCGEr3l0jq5Sqj5QDfg6t01rfRL4AWiX09QGCMnXJxXYm6fPDcDx3CI3x3+xX9W4\n3pMxCyGE0XhwrBVCCFML8fD+qmEvRg/naz+csw2gKnA+Z1AurE814EjejVrrS0qpY3n6OFFKXQXc\nBvwJnHUzfiGEuJIwoB6wUmt91I9xeGqsdSLjqBDCB3w6jnq60PWn24D3/R2EECIo3IP9PgKzkXFU\nCOErPhlHPV3oHgIU9isJea80VAW25ulTVilVMd+Vhqo523L75F+FoQxQOU+f/P4EeO+992jWrFkp\nUvCPMWPGMG3aNH+H4RaJ3bsmrJ3AzuM7mdxlslP7i8+8yDMvPuOnqErH3djDQsKIiYjxQkTFk5KS\nwr333gs5440feWqsze9PMO44WlxG+L4vDcnP+MycY8qXX3Lv+PHgo3HUo4Wu1nq3UuoQ9pUSfgbH\nDRLXA7NzutmAizl98t4gUQf4LqfPd0CUUuraPPN0u2Af2H8o5OPPAjRr1oz4+HhPpuUTlSpVMmTc\nILF7W1pyGjc0uoE7O9/p1D6/ynyXNqMwcuw5/PprfQ+OtfkZehwtLiN835eG5Gd8ps4xJSX3bz4Z\nR0tc6OasZdsIe9EJ0EAp1Ro4prX+C/tyNs8qpf7AXq1PAvYBn4D9hgml1HxgqlLqOHAKmAF8m3sX\nsNZ6u1JqJfBvpdTDQFlgJrDYrCsuHDpk3LQkdu/RWpOansrfmv7NZZunYv/jD6hbF0JDPbK7Yilp\n7JmZ8MUXEBICvXt7KagA44uxNlgF+vd9aUl+xhcMOfqKO6sutMH+qzEb9psh/gkkA88DaK2nYC9K\n38B+9bU8cLvW+nyefYwBPgeWAt8AB7Cv85jXAGA79tUWPgfWA8PciNcQ9u/f7+8Q3Caxe8/hzMNk\nnMsg7qo4l22eiP3sWWjdGp59ttS7KpHixJ6ZCR99BHffDTEx0Lcv/Oc/PggucPhqrA06gf59X1qS\nn/EFQ46+UuIrulrrdRRRIGutk4CkK2w/B4zIeRXW5wRwb0njM6qEhAR/h+A2id17UtNTAWhapanL\nNk/EnpwMWVkwcyaMGQPVCrwXv2BffQXNmtmvBpdUQbEfPAg//ACbN9tf331njy0hAZKS4K67oEGD\nkn+WUflqrA1Ggf59X1qSn/EFQ46+YqZVFwytf//+/g7BbRK796QeTaWMKkPDyg1dtnki9h9+gLAw\nKFsWJk+G6dOL976//oKePWH0aHj11ZJ/bm7sWVnw5pswYwbs3m3fVq0aXH+9vbjt3RsauqYuRKkE\n+vd9aUl+xufrHPfu3Ut6erpPPmt37mDvI6V6MlogUUrFAzabzWbeCdwi6Dy28jE+//1zfh/xu1f2\n37cvHDgAXbvCiy/a5+vWqlWMuB6DadOgTx/44IOSf+7JkzBnDkydCseOwT33wJ13Qtu2ULMmKFX0\nPvwhOTk590pLgtY62d/xeJqMo0IEn7179xIX14yzZ7N88nlxgP13lb4ZR+WKrgH58n9ewr82/7iZ\naqoaycneGQvWr4dbb4WbboLXXrNfoX366Su/JyMD5s4Fi8V+82xJQjtxwl4YL1pknx98551w//32\n4hbgyBH7y5+qVKlCnTp1/BuEEH4mP2eCR0pKSk6ROwmo76G9RgHVC9k2D5jroc8pBq21KV5APKBt\nNps2okGDBhWr3549e3R4eLjGfnOKvOQlLw+/wsPD9Z49ewr8/rPZbLn94nUAjHuefmHwcbS4ijve\nGlVp85OfM/Iq7ctiCdewR4N2ecUxKbefT8ZRuaIbILp27Vqsfunp6WRlZZl+QXch/CH3gRDp6ely\nVdfEijveGlVp85OfM6I0Lj9YJx37st3+JYVugCjpxHOzL+guhBDeYvabmTyVn/ycEWbgzjq6Qggh\nhBBCBDwpdIUQQgghhClJoRsgNm7c6O8QhBAiKJh9vDV7fkKUhBS6AWLKlCn+DkEIIYKC2cdbs+cn\nRElIoRsglixZ4u8QhLiidevWYbFYWL9+vb9DEaJUzD7emj0/4b6FCxdisVjYu3evo61Tp07cfPPN\nfozKu6TQDRDh4eH+DsHv3n77bSwWCxaLhU2bNhXYp3bt2lgsFqxWq4+jEwAqUB9ZJkQJmH28NXt+\nomgvv/wyn3zyiUu7UsplHDf7uC6Frgg45cuXZ9GiRS7t69atY//+/YSFhfkhKiGEEMIYXnrppQIL\n3YEDB3LmzJmgWidcCl0RcLp3785HH31Edna2U/uiRYto06YN1apV81Nk7tNac+7cOX+HcUWXLl3i\nwoUL/g5DCCEM78yZM/4OoUBKKcqWLevvMHxKCt0AMXbsWH+HEBCUUvTv35+jR4+yevVqR/uFCxdY\nunQpAwYMyH1UqROtNdOnT6dly5aUL1+eatWq8dBDD3HixAmnfp9++ik9evSgZs2ahIWF0ahRI154\n4QWXorpTp060atWKlJQUOnfuTEREBLVq1eLVV18tVh4Wi4WRI0eyaNEiWrZsSVhYGCtXrix2rI8/\n/jhVqlRx2ueIESOwWCzMmjXL0XbkyBEsFgtvvPGG4zhNmDCBNm3aEBUVRWRkJB07duSbb75x2tee\nPXuwWCxMnTqVf/3rXzRq1IiwsDBSUlIA2L9/P7169SIyMpKqVavy2GOPce7cOZdj/8cff/D3v/+d\n6tWrU758eWrXrk3//v05depUsY6TEP5g9vHW7PmVVlJSEhaLhdTUVPr06UOlSpWoUqUKo0ePdrkg\n8d5779GmTRvCw8O56qqr6N+/P/v27XPqk/vzIjk5mY4dOxIREcEzzzzj2L5ixQpuuukmKlasSKVK\nlWjbti2LFy922scPP/xAt27diIqKIiIigk6dOrlM4cuNe+fOnQwaNIjo6GiioqIYMmQIZ8+edfSz\nWCxkZWU55uNaLBaGDBkCFDxHtyDnz5/nueeeo3HjxoSFhVGnTh2eeOIJzp8/X/wDHSDkyWgBIph+\njVCUevXqccMNN7B48WJuu+02AL788ktOnjxJv379+Ne//uXynqFDh/LOO+8wZMgQRo0axe7du5k5\ncybbtm3j22+/pUyZMoD9m7xChQo8/vjjREZGsmbNGiZMmMCpU6d45ZVXHPtTSnHs2DFuv/12evfu\nTb9+/Vi6dClPPvkkrVq1csR1JV9//TUffvghw4cPp0qVKtSrV6/YsSYmJjJ9+nR+++03mjdvDtiX\nDCpTpgwbNmxg+PDhAKxfvx6lFB07dgTg5MmTvPXWW/Tv35+hQ4dy6tQp5s+fT7du3di8eTOtWrVy\nivGtt97i3LlzDBs2jHLlylG5cmXOnj3LzTffzL59+xg1ahTVq1fn3XffZc2aNU5zuS5cuEDXrl25\ncOECI0eOpFq1auzfv5/PP/+cEydOUKFCheL+kwvhU2Yfb82eX2nljmN9+vShfv36TJ48me+//54Z\nM2Zw4sQJFi5cCMCLL77IhAkT6NevHw8++CBpaWnMmDGDm266ia1bt1KxYkXH/tLT0+nevTv9+vVj\n4MCBVK1aFbD/zHnggQdo2bIlTz/9NFFRUWzdupWVK1c6nmC3Zs0aunfvTps2bRzF7IIFC7j55pvZ\nuHEjbdq0cYm7QYMGTJ48meTkZObNm0fVqlV5+eWXAXtx/sADD3D99dczdOhQABo2bOjYR1FzcrXW\n9OzZk02bNjFs2DCaNm3KL7/8wrRp09ixYwfLli3z1D+Fb2itTfEC4gFts9m0mdlsNm3WPBcuXKgt\nFou22Wx69uzZulKlSvrs2bNaa6379Omju3TporXWul69erpnz56O923YsEErpfSSJUuc9rdq1Sqt\nlNKLFy92tOXuL6+HHnpIR0ZG6vPnzzvaOnXqpC0Wi37//fcdbefPn9fVq1fXd999d5G5KKV0SEiI\n3r59u1N7cWNNS0vTSin9+uuva621zsjI0GXKlNF9+/bV1atXd7xv1KhRukqVKo6vs7Oz9YULF5z2\nnZGRoatVq6b/8Y9/ONr+/PNPrZTSUVFR+ujRo079p0+fri0Wi/74448dbWfOnNGNGzfWFotFr1u3\nTmut9bZt27RSSi9btqzI42EURX1/5W4H4nUAjHuefgXLOCquzMw/Z7TWOikpSSul9N/+9jen9kcf\nfVRbLBb9yy+/6D179uiQkBA9efJkpz7/+9//dGhoqH755Zcdbbk/L/7973879c3IyNAVK1bU7du3\n1+fOnSs0niZNmuju3bs7tZ09e1Y3aNBA33bbbS5xP/jgg059e/furWNiYpzaIiMj9eDBg10+K/fn\n7J49e5zi79y5s+Prd999V4eEhOhNmzY5vfeNN97QFotFf/fdd4XmonXecdKmQbu84pjk03FUruia\nXFYWbN/u3c9o2hQ8fZNvnz59GD16NJ9//jm33XYbn3/+udOv7PNaunQpUVFRdOnShaNHjzrar732\nWiIjI1m7di39+vUDoFy5co7tp0+f5ty5c3To0IE333yT7du3c/XVVzu2R0ZGMmDAAMfXoaGhtG3b\nll27dhUrh06dOhEXF+dWrFWqVKFp06asX7+eYcOGsXHjRkJCQhg7diwffvghO3fupGHDhmzYsIEO\nHTo49qOUIiTE/m2ttebEiRNcunSJNm3akJyc7BLjXXfdReXKlZ3aVqxYQfXq1endu7ejLSwsjKFD\nh/LEE0842ipVqgTAV199Rbdu3ShfvnyxjosQwmQM+INGKcWjjz7q1DZixAjmzJnDl19+SdmyZdFa\nc/fddzuN1bGxsTRu3Ji1a9fy5JNPOtrLlSvHoEGDnPa3evVqTp8+zZNPPlnovNht27axY8cOxo8f\n7/Q5Wmu6dOnCe++95xL3sGHDnNoSExNZvnw5p0+fJjIyskTHoSBLly6lWbNmNGnSxCmmzp07o7Vm\n7dq13HDDDaX+HF+RQtfktm+HhATvfobNBvHxnt1nlSpVuOWWW1i0aBGZmZlkZ2dz1113Fdh3x44d\nnDhxgtjYWJdtSimOHDni+Pq3337jmWeeYe3atZw8edKpX0ZGhtN7a9Wq5bK/6Ohofvnll2LlkDtV\nwd1YExMTWbFiBYDj11cJCQlUrlyZDRs2EBsby08//cQ999zjtJ+3336bqVOnsn37dqebyxo0aFCs\nGPfs2UOjRo1c2vMX7fXq1ePxxx9n6tSpvPfeeyQmJmK1Wrn33nsdv9ITQgQBg/6gyT/ONWzYEIvF\nwp9//olSiuzs7ALHwoJu6KpZs6bjIkOunTt3AtCiRYtCY9ixYwdgXw2hIBaLhYyMDMeFBXCdmhId\nHQ3A8ePHPVLo7tixg+3btxMTE+OyLf/PKSOQQjdAbN++naZNm3p8v02b2scHb/JC2AAMGDCABx98\nkIMHD3L77bcXOuczOzubqlWrsmjRotxfvzrJ/WbNyMigY8eOREVF8cILL9CgQQPCwsKw2Ww8+eST\nLjek5c7rza+gzyhIQVc4ixsrQIcOHZg3bx67d+9mw4YNJCYmOto3bNhA9erV0Vo72sE+N2vw4MH0\n7t2bcePGERsbS5kyZXjppZcKvBJd2quwr776KoMGDeKTTz5h1apVjBw50jHfrUaNGqXatxDe4q3x\nNlD4PD8j/6DJI+/c1ezsbCwWC1999RUWi+t9+/kLSnfH0tyfO//85z9p3bp1gX3yf1ZpfzYVJ6ar\nr76aadOmFbjP2rVre+RzfEUK3QAxbtw4Pv30U4/vNzzc81dbfeVvf/sbw4YN44cffuCDDz4otF/D\nhg35+uuvad++vdPUhPy++eYbjh8/zieffMKNN97oaM/9X7cvFDdWwFHArl69mi1btvDUU08B0LFj\nR+bOnUv16tWJiIggIc+VlI8//piGDRuydOlSp31NmDCh2DHWrVuX//3vfy7t2wv51WSLFi1o0aIF\nTz/9NN9//z3t27fn9ddfZ+LEicX+TCF8yVvjbaDweX4G/UGzY8cO6tat6/j6jz/+IDs7m3r16mGx\nWNBaU69evQKv6hZHw4YN0Vrz66+/Fvgbtdw+ABUqVPDo08lK8xCIhg0b8vPPP9O5c2ePxeNPsrxY\ngChs/mkwi4iI4PXXXycpKYmePXsW2q9Pnz5cvHixwMLq0qVLjikJZcqUQWvtdOX2/PnzzJkzx/PB\nlzJWsE8NqFGjBtOmTePixYuO4jwxMZGdO3eydOlSbrjhBqerDQX9T/+HH37gu+++K3aM3bt358CB\nA3z88ceOtqysLP7973879Tt16hSXLl1yamvRogUWiyXg1wwWwc3s463Z8/MErTWzZ892apsxYwZK\nKbp3707v3r2xWCw8//zzBb7/2LFjRX5G165dqVChAi+//HKhY2JCQgINGzbktddeIzMz02V7enp6\nMbJxFRER4bK8ZnH16dOHffv2uYz5AGfPniUrK8ut/fqLXNENELIcjF3+X5Pcd999Rb6nY8eODBs2\njMmTJ7Nt2za6du1KaGgov//+O0uXLmXGjBn07t2b9u3bEx0dzcCBAxk5ciRg/1W/Lx9/WNxYcyUm\nJrJkyRJatWrlmKMVHx9PREQEO3bscJmf26NHD5YtW0avXr2444472LVrF2+88QYtWrTg9OnTxYrx\nwQcfZNasWdx33338+OOPjuXFIiIinPqtWbOG4cOHc/fdd9OkSRMuXrzIO++8Q0hICH//+99LeaSE\n8B6zj7dmz89Tdu/ezZ133km3bt3YtGkT77//Pvfeey8tW7YE4IUXXuDpp59m9+7d9OrViwoVKrBr\n1y6WL1/OsGHDeOyxx664/woVKjBt2jQefPBBrrvuOgYMGEB0dDQ//fQTZ86cYcGCBSilmDdvHt27\nd6dFixYMHjyYmjVrsn//ftauXUulSpUKfMJZURISEvjvf//LtGnTqFGjBvXr16dt27bFeu99993H\nhx9+yMMPP8zatWu58cYbuXTpEikpKXz00UesWrWKeANdwZdCVwSU4hSdBa0DOHfuXNq0acMbb7zB\nM888Q0hICPXq1WPgwIGOK6GVK1fmiy++4PHHH2f8+PFER0dz3333cfPNNxe4Lm5hsbgbY0lizZWY\nmMgHH3zgNA+3TJkytGvXjjVr1ji1AwwaNIjDhw/zxhtvsGrVKpo3b87777/Phx9+yPr164sVY/ny\n5VmzZg0jRoxg1qxZhIeHc++999KtWze6devm6Ne6dWu6devG559/zv79+wkPD6d169Z89dVXxR5Q\nhRDCH5RSfPDBB4wfP56nnnqKkJAQRo4cyZQpUxx9nnjiCeLi4pg2bZrjt3C1a9emW7duWK1Wl/0V\nZMiQIVStWpXJkyfzwgsvEBoaStOmTRkzZoyjz0033cR3333HpEmTmD17NqdPn6ZatWpcf/31Liss\nFNfUqVMZNmwY48eP58yZM9x///1XHJfzxq+U4pNPPmHatGm88847LF++nPDwcBo0aMCYMWNo0qSJ\nWzH5i/LU5GV/U0rFAzabzWao/2mUVHJyMgkJCZg9TyH8oajvr9ztQILW2nW9NoMLlnFUXJnZf848\n//zzTJw4kbS0NJflFUXpXR4nbdiX5nYWxwukMh58NI7KHN0AkfepXEIIIbzH7OOt2fMToiSk0A0Q\nRpvcLYQQRmX28dbs+QlRElLoBojC7uwUQgjhWWYfb82enxAlIYWuEEIIIYLGc889x6VLl2R+bpCQ\nQlcIIYQQQpiSFLoBwt1FoYUQQpSM2cdbs+cnRElIoRsghgwZ4u8QhBAiKJh9vDV7fkKUhBS6ASIp\nKcnfIQghRFAw+3hr9vyEKAl5MlqAMOOi3EIIEYjMPt56Kr+UlBSP7EcEl0A7b6TQFUIIIYRDlSpV\nHI/+FsIdFks42dlV/B0GIIWuEEIIIfKoU6cOKSkpclNbkEhJScn5T817QDOP7NNe5NbxyL5KSwrd\nADF//nweeOABf4chfMRisZCUlMSECRPcen+nTp04duwYP//8s4cjc8/bb7/N4MGD+fHHHz32a9NB\ngwaxbt06du/efcV+e/bsoX79+ixcuJCBAwd65LOFuZl9vPVEfnXq1KFOncAoVPIz+78f+CvHZoD5\npvXIzWgBIjk52d8h+N3bb7+NxWJxvEJDQ6lVqxaDBw/mwIED/g7P5w4ePMjzzz9fYDGrlPJDRFfm\n6ZiUUgGZpzA+s4+3kp/xBUOOviJXdAPE7Nmz/R1CQFBKMWnSJOrVq8fZs2f5/vvvWbBgAd9++y2/\n/vorZcuW9XeIPnPgwAGef/556tevT6tWrfwdjhCmYfbxVvIzvmDI0Vek0BUBp1u3bo5ffw8ZMoSr\nrrqKKVOm8Omnn3LXXXf5ObriOXv2LGFhYaXah9baQ9E4O3fuHGXLlpWrpUIIIUxPpi6IgJeYmIjW\nmp07d7psW7FiBR07diQyMpKKFSvSo0cPfvvtN8f2zz77DIvFwq+//upoW7ZsGRaLxaVobtasGf37\n93d8vWDBArp06ULVqlUJCwujRYsWvP766y4x1KtXD6vVyqpVq7juuusoX748b775JgDnz59nzJgx\nxMbGUrFiRXr16sX+/fuLzHndunW0bdsWpRSDBg3CYrFQpkwZ3nnnHad+KSkpdO7cmYiICGrVqsWr\nr77qsh+LxcIHH3zAs88+S61atYiIiODUqVMAZGRkMHr0aOrUqUNYWBiNGzdmypQpLkX2kiVLaNOm\nDRUrVqRSpUq0atWKGTNmuMR97tw5HnvsMWJjY4mMjKR3794cPXrUpd+cOXNo2bIlYWFh1KxZk+HD\nh5ORkVHkccnIyGDQoEFERUURHR3N4MGDOXHihEu/w4cPM3jwYGrXrk1YWBg1atSgV69e7N27t8jP\nEEIIYR5yRVcEvNybkaKjo53a3333XQYNGkS3bt2YMmUKWVlZzJ07l8TERLZu3UqdOnXo0KEDSinW\nr19Py5YtAdiwYQMWi4WNGzc69pWenk5qaiqjRo1ytL3++uu0bNmSO++8k5CQED777DMeeeQRtNY8\n/PDDjn5KKbZv386AAQMYNmwYQ4cOJS4uDoAHHniARYsWcc8999CuXTvWrFnDHXfcUeTV1GbNmjFx\n4kQmTJjAsGHDSExMBKB9+/aOPseOHeP222+nd+/e9OvXj6VLl/Lkk0/SqlUrbrvtNqf9TZo0iXLl\nyjF27FjHFd0zZ87QsWNHDh48yEMPPUTt2rXZtGkTTz31FIcOHWLq1KkArF69mgEDBnDrrbcyZcoU\nwF5gb9q0iZEjRzo+Q2vN8OHDqVy5MklJSfz5559MmzaN4cOHs3jxYke/pKQkJk6cSNeuXXnkkUdI\nTU1lzpw5/Pjjj3z77beUKVOm0ONitVrZtGkTDz/8ME2bNuU///kP999/v8vx7N27NykpKYwcOZK6\ndety5MgRVq9ezd69ewP2BhshhBBeoLU2xQv7rYLaZrNpI+rZs2ex+tlsNm3kPK9k4cKF2mKx6DVr\n1uj09HS9b98+vXTpUh0bG6vDw8P1/v37HX1Pnz6to6Oj9UMPPeS0jyNHjuioqCg9bNgwR1vLli11\nv379HF8nJCTovn37aovFolNTU7XWWi9btkxbLBb9yy+/OPqdPXvWJcZu3brpRo0aObXVq1dPWywW\nvXr1aqf2n376SSul9IgRI5za77nnHm2xWPTzzz9/xePx448/aqWUfvvtt122derUSVssFv3+++87\n2s6fP6+rV6+u7777bkfbN998o5VSulGjRvrcuXNO+5g0aZKuUKGC3rlzp1P7U089pUNDQ/W+ffu0\n1lqPHj1aR0VFXTHWhQsXaqWUvu2225zaH3vsMR0aGqpPnjyptdY6LS1NlytXTt9+++1O/WbPnq0t\nFoteuHCho23QoEG6fv36jq+XL1+ulVL6n//8p6MtOztbd+zYUVssFsdxOnHihEu/4irq+yt3OxCv\nA2Dc8/TL6ONocRV3vDUqyc/4fJnj5XHNpkF7/RXHJJ+Oo3JFN0AMHz7cK/vNupDF9vTtXtl3rqZV\nmhIeGu6RfWmt6dKli1Nb/fr1WbRoETVq1HC0rV69moyMDPr16+f0q3GlFNdffz1r1651tCUmJvLp\np58CcOrUKX766SemTJnCmjVr2LBhA02aNGHDhg1ERUU5rvoClCtXzvH3kydPcuHCBTp27MiqVas4\ndeoUFSpUcIrxlltucYr7yy+/RCnFiBEjnNpHjx7NokWL3Dk8TiIjIxkwYIDj69DQUNq2bcuuXbtc\n+g4aNMjlRr6lS5eSmJhIpUqVnI5hly5dmDx5MuvXr6d///5ERUWRmZnJypUrXa4U56WUYujQoU5t\niYmJTJ8+nT179tCyZUv++9//cuHCBUaPHu3U78EHH+Tpp5/miy++4P777y9w/ytWrCA0NJSHHnrI\n6TNHjBjBhg0bHG3ly5enbNmyfPPNNwwZMoSoqKhCYxbByVvjbaCQ/IwvGHL0FSl0A0TXrl29st/t\n6dtJeDPEtBuZAAAgAElEQVTBK/vOZRtqI766Z9beU0oxZ84cGjduTEZGBm+99Rbr1693KdJ27NiB\n1prOnTsXuI9KlSo5vk5MTOSNN95g165d7NixA4vFQrt27UhMTGTDhg088MADbNy4kRtvvNFpP99+\n+y3PPfcc33//PVlZWU77z8jIcCl089uzZw8Wi4WGDRs6tedOayitWrVqubRFR0fzyy+/uLTXq1fP\npW3Hjh388ssvxMTEuGxTSnHkyBEAHnnkET766CO6d+9OjRo16Nq1K3369Cmw6K1du7ZLPADHjx8H\n7McEoEmTJk79QkNDadCggWN7Qfbs2UP16tUJD3f+T1X+41m2bFleeeUV/u///o+qVatyww030KNH\nDwYOHEjVqlUL3b8IHt4abwOF5Gd8wZCjr0iha3JNqzTFNtTm9c/wpOuuu86x6sKdd95Jhw4dGDBg\nAKmpqY4iJzs7G6UU7733XoHFS0jI5VO7Q4cOaK1Zv349O3fuJD4+nvLly5OYmMjMmTPJzMxk69at\nvPTSS4737Nq1i1tuuYVmzZoxbdo0ateuTdmyZfniiy+YPn062dnZTp9Xvnx5jx6D4ihsLqvWrqs1\nFBRfdnY2t956K0888USB78ktRmNiYti2bRsrV65kxYoVrFixggULFnD//fezYMGCImPK/fWRL40a\nNQqr1cry5ctZuXIlEyZM4OWXX2bt2rW0bt3ap7EIIYTwHyl0TS48NNxjV1v9wWKx8PLLL9O5c2dm\nzZrFuHHjAGjYsCFaa2JiYrj55puvuI/atWtTp04d1q9fz65duxw3dnXs2JHHH3+cjz76iOzsbDp2\n7Oh4z2effcb58+f57LPPqFmzpqP966+/LnbsdevWJTs7m507d9K4cWNH+/btxZtK4u3lvxo2bMjp\n06cLvCqeX0hICHfccQd33HEHAA8//DBvvvkm48ePp0GDBld8b9486tatC0BqaqrTVeYLFy6we/du\nbr311kL3U7duXdasWUNWVpbTVd3Cjmf9+vUZM2YMY8aMYefOnbRu3Zp//vOfLitXCCGEMC9ZXixA\nLF++3N8hBKybbrqJtm3bMn36dM6fPw/AbbfdRsWKFXnppZe4ePGiy3vyP6M9MTGRNWvWsGXLFkeh\ne8011xAZGcnkyZMpX748CQmXp3jkXpnMe+U2IyODhQsXFjvu22+/Ha21yzJc06dPL1YRGxERAVDg\n8lme0KdPH7777jtWrVrlsi0jI4NLly4B9tUd8rv66qsB+3JiJXHLLbcQGhrqckzmzZvHyZMn6dGj\nR6Hv7d69OxcuXGDu3LmOtuzsbGbOnOl0PM+cOeMSV/369alQoUKJ4xXmZPbxVvIzvmDI0Vfkim6A\nWLx4Mb169fJ3GH5X2K+4x44dy913383ChQsZOnQoFSpUYO7cuQwcOJD4+Hj69etHTEwMe/fu5Ysv\nvqBDhw5OxVRiYiLvv/8+FouFDh06AParxe3bt2flypV07tzZabpD165dCQ0NpUePHgwbNoxTp04x\nb948qlatyqFDh4qVS+vWrenfvz9z5szhxIkTtG/fnq+//pqdO3cW61f5DRs2JCoqitdff53IyEgi\nIiK44YYbHFdFS2vs2LF8+umn9OjRg0GDBpGQkEBmZiY///wzy5Yt488//6Ry5cr84x//4NixY9x8\n883UqlWLP//8k1mzZnHttdfSrFkzx/4Kyylve5UqVXjqqaeYOHEi3bp1w2q1sn37dubOnUvbtm25\n5557Co23Z8+e3HjjjTz55JPs3r2b5s2bs2zZMseawLl+//13unTpQp8+fWjevDkhISEsW7aMI0eO\nOK2TLIKX2cdbyc/4giFHn/HF0g6+eBEky+IEw/JiBeWWnZ2tGzVqpBs3bqyzs7Md7evWrdO33367\njo6O1uHh4bpx48Z6yJAhOjk52en9v/32m7ZYLLply5ZO7S+++KK2WCw6KSnJ5TM///xzfc011+jw\n8HDdoEED/dprr+kFCxZoi8Wi9+zZ4+hXv359bbVaC8zp3LlzevTo0TomJkZXqFBB9+rVS+/fv19b\nLBY9ceLEIo/JZ599plu2bKnLli3rtIRWp06ddKtWrVz6Dxo0SDdo0MDx9TfffKMtFov++OOPC9x/\nZmamfuaZZ3STJk10WFiYjo2N1R06dNDTpk3TFy9e1Frbl17r1q2brlatmg4LC9P16tXTjzzyiD58\n+LBjP4X92+V+/rp165za58yZo5s3b67LlSunq1evrocPH64zMjKumIvWWh8/flzff//9OioqSkdH\nR+tBgwbpn376yenYHD16VI8YMUI3b95cV6hQQUdHR+t27doVegzykuXFgmMcFUJcZvblxZTWvr1J\nxFuUUvGAzWazOW5kMqPk5GQSEhIwe55C+ENR31+524EErXWyzwP0smAZR4UQl10e12zY/6/rXXG8\nQCrjwUfjqMfn6CqlLEqpSUqpXUqpLKXUH0qpZwvoN1EpdSCnz2qlVKN828sppWYrpdKVUqeUUkuV\nUrGejlcIIYzIU2OtEEKYmTduRnsSGAY8AjQFxgHjlFKO1Y+VUk8Aw4GhQFsgE1iplMq7WOp04A7g\n70BHoAbwsRfiFUIII/LUWCuEEKbljUK3HfCJ1vorrfVerfUyYBX2QTbXKGCS1vpzrfWvwEDshWwv\nAKVURWAIMEZrvU5rvRUYDNyolMq7H9MYPHiwv0MQQhhLqcfaYGX28VbyM75gyNFXvFHobgK6KKUa\nAyilWgM3Al/mfF0fqAY4FiTVWp8EfsA+cAO0wb4iRN4+qcDePH1MRZ6CIoQoIU+MtUHJ7OOt5Gd8\nwZCjr3hjebHJQEVgu1LqEvZi+hmt9ZKc7dWw3213ON/7DudsA6gKnM8ZlAvrYyqy7JEQooQ8MdYG\nJbOPt5Kf8QVDjr7ijSu6fYEBQD/gWuB+YKxS6j4vfJaL7t27Y7VanV7t2rVzWXx51apVWK1Wl/c/\n+uijzJ8/36ktOTkZq9Xq8hCC5557jldeecWpbe/evY61QfOaOXMmY8eOdWrLysrCarWyceNGp/bF\nixcX+GuLvn37snbt2kIyF0J40uLFi7FarcTFxdG8eXOsVitjxozxd1h5eW2sNfs4KnlIHpJHwXnY\nh5X8D6tYBbjmAY8C8/O1Jef0zc1jcc7XcUBzwMo+3itgX17k6fXKsE8veDhf2zPAbzl/rw9kA63y\n9fkGmJbz987AJaBivj5/AqMK+dygWP/RzOvoCuFvRlpH1xNjbQH7DIpxVAhxmdnX0fXGFd3wnCI1\nr2xyrh5rrXcDh4AuuRtzbj67HvucM7Av5nYxX584oA7wnRdi9rv8/9sSQogieGKsDUpmH28lP+ML\nhhx9xRtzdD8DnlVK7QP+h/0KwRhgXp4+03P6/IH9Ku0kYB/wCdhvmFBKzQemKqWOA6eAGcC3WuvN\nXojZ76ZMmeJ4NG1xpKSkeDEaIYKTwb6vSj3WBquSjrdGI/kZXzDk6CveKHSHYx9MZwOxwAFgbk4b\nAFrrKUqpcOANIArYANyutT6fZz9jsF+tWAqUA77CPiHElJYsWVJ0J6BKlSqEh4dz7733ejkiIYJT\neHg4VapU8XcYxeGpsTboFHe8NSrJz/iCIUdf8Xihq7XOBB7LeV2pXxKQdIXt54AROS/TCw8PL1a/\nOnXqkJKS4jJhXRhb13e7clfzuxiaMNRl28qV8PTTsGYNVKrkh+CCTJUqVahTp46/wyiSp8baYFTc\n8daoJD/jC4YcfcUbV3SFl9WpU8cQP4hF8WitOfnFSVpd04r4eNfnjH/0EdSoAZ07+yE4IYQQwsC8\ncTOaEKIETp8/zYXsC1xV/qoCt2/eDG1N+TxAIYQQwruk0A0Q+de+MxKJvXTSs+zTUK4Kdy10s7Nh\ny5aCC91AiN1dRo5dGJ/Zzz/Jz/iCIUdfkUI3QBh5KoLEXjpHzxwFKPCKbmoqnDpVcKEbCLG7y8ix\nC+Mz+/kn+RlfMOToK0rbFwk3PKVUPGCz2WwFznMUIlCt/GMl3d7vxt7Re6ldqbbTtnfegfvvhxMn\n5Ea0QJCcnExCQgJAgtY62d/xeJqMo0IEn8vjmg37KoXeFccLpDIefDSOyhVdIfzMcUW3gKkLmzdD\n06ZS5AohhBDukEJXCD9Lz0onLCSM8FDX5WTkRjQhhBDCfVLoBojt27f7OwS3SeylczTraIHzc8+d\ng23bCi90AyF2dxk5dmF8Zj//JD/jC4YcfUUK3QAxbtw4f4fgNom9dI6eOVrgtIWffoILFwovdAMh\ndncZOXZhfGY//yQ/4wuGHH1FCt0AMWvWLH+H4DaJvXSOnjlKlXDXR85u3gyhodCqVcHvC4TY3WXk\n2IXxmf38k/yMLxhy9BUpdAOEkZcSkdhLJz0rvcCpC5s3wzXXQLlyBb8vEGJ3l5FjF8Zn9vNP8jO+\nYMjRV6TQFcLPCpujKzeiCSGEEKUjha4QflbQHN0TJ+wPi5BCVwghhHCfFLoB4pVXXvF3CG6T2Evn\naJbrHN0ff7T/eaVCNxBid5eRYxfGZ/bzT/IzvmDI0Vek0A0QWVlZ/g7BbRK7+85ePEvmhUyXqQub\nN0PFitCkSeHv9XfspWHk2IXxmf38k/yMLxhy9BV5BLAQfrT/5H5qTavFFwO+oHvj7o72Xr3g9Gn4\n73/9GJxwIY8AFkKYjTwCWAjhNY7H/xZwRVfm5wohhBClI4WuEH50NMte6Oado7t/Pxw8KIWuEEII\nUVpS6AaI9PR0f4fgNom9eJb+tpQHP32QbJ19+fOz7J+fd9WFzZvtfxZV6MpxF8I9Zj//JD/jC4Yc\nfUUK3QAxZMgQf4fgNom9eN79+V3mbZ3Hv77/l6Pt6JmjlFFlqFSukqNt82aoWRNq1Ljy/uS4C+Ee\ns59/kp/xBUOOviKFboBISkrydwhuk9iLx3bAxlXlr+LJr59k26FtgH3qQuXylVFKOfoVd36uHHch\n3GP280/yM75gyNFXpNANEEa+w1liL9rh04fZf2o//+r2L5pWacqAjweQdSGLo2ec19DNzoYtW4pX\n6MpxF8I9Zj//JD/jC4YcfUUKXSF8wHbQBkD72u1Z1HsRu0/sZtzqcS5PRduxA06dgjZt/BWpEEII\nYR4h/g5AiGBgO2AjOiyaelH1UErx2q2vMXzFcKpGVOWGWjc4+qWm2v9s3txPgQohhBAmIld0A8T8\n+fP9HYLbJPai2Q7aSKiR4JiL+8h1j9C9cXcOZx52WkN3xw6IiIDq1Yvepxx3Idxj9vNP8jO+YMjR\nV6TQDRDJycZ9yJLEXjTbQRsJ1RMcXyulWHDnAqpFVqNeVD1H+++/Q+PGkOfetELJcRfCPWY//yQ/\n4wuGHH1FHgEshJcdyTxC1deq8uFdH3J3i7udtp08d5Lw0HBCLPZZRDffDDEx8MEH/ohUFEUeASyE\nMBt5BLAQolRsB+w3oiXUSHDZVrFcRUeRC5ev6AohhBCi9KTQFcLLbAftN6LVj6p/xX6ZmfbH/zZp\n4qPAhBBCCJOTQlcIL7MdtBFfPd7poRAF+eMP+59yRVcIIYTwDCl0A4TVavV3CG6T2K/MdsD5RrTC\n/P67/c/iXtGV4y6Ee8x+/kl+xhcMOfqKFLoBYvjw4f4OwW0Se+HSMtP46+RfBc7PzW/HDoiOhquu\nKrIrIMddCHeZ/fyT/IwvGHL0FSl0A0TXrl39HYLbJPbC5T4RrbhXdEsyP1eOuxDuMfv5J/kZXzDk\n6CtS6ArhRbYDNqLComgQ3aDIvjt2yI1oQgghhCdJoSuEFxX3RjSQpcWEEEIIT5NCN0AsX77c3yG4\nTWIvXP4nohXm+HFITy/ZFV057kK4x+znn+RnfMGQo69IoRsgFi9e7O8Q3CaxFyw9K529GXuLVeju\n2GH/syRXdOW4C+Ees59/kp/xBUOOviKFboD4wMDPfJXYC3alJ6Lll7u0WEkKXTnuQrjH7Oef5Gd8\nwZCjr0ihK4SX2A7aqFSuEg2jGxbZd8cOqFYNKlTwQWBCCCFEkJBCVwgvKemNaLLighBCCOFZUugK\n4SW2Azba1GhTrL47dsiKC0IIIYSnSaEbIAYPHuzvENwmsbs6mnWUPRl7inUjmtbuXdGV4y6Ee8x+\n/kl+xhcMOfqKFLoBwshPQZHYXf1y5BcAWldrXWTfI0fg1KmSX9GV4y6Ee8x+/kl+xhcMOfqKFLoB\non///v4OwW0Su6vDpw8DUKNCjSL75q64UNIrunLchXCP2c8/yc/4giFHX5FCVwgvSMtKo2yZslQo\nW/QyCr//DkpBw6IXZxBCCCFECUihK4QXpGWmERMeU6wVF3bsgDp1ICzMB4EJIYQQQUQK3QCxceNG\nf4fgNondVVpWGjERMcXq6+7SYnLchXCP2c8/yc/4giFHX5FCN0BMmTLF3yG4TWJ3lZZlv6JbHO4u\nLSbHXQj3mP38k/yMLxhy9BUpdAPEkiVL/B2C2yR2V0cyj7hc0U1NhY8/du6XnQ1//OHeFV057kK4\nx+znn+RnfMGQo69IoRsgwsPD/R2C2yR2V2mZacSGxzq1TZoEd90FCxZcbtu3D86ede+Krhx3Idxj\n9vNP8jO+YMjRV6TQFcILCpqju3kzVKoEQ4fC11/b29xdWkwIIYQQRfNKoauUqqGUelcpla6UylJK\n/aSUis/XZ6JS6kDO9tVKqUb5tpdTSs3O2ccppdRSpZTzJTIhAtCl7EsczTrqNEf3xAn7XNxp06BL\nF+jdG/73P3uhGxIC9er5L15hXJ4Ya4UQwsw8XugqpaKAb4FzwG1AM+Bx4HiePk8Aw4GhQFsgE1ip\nlCqbZ1fTgTuAvwMdgRpAvhmO5jF27Fh/h+A2id3ZsTPH0GinK7o//mj/s107+PBDqFsXuneHDRug\nQQN7sVtSctyDmwfH2qBj9vNP8jO+YMjRV9z48VqkJ4G9Wut/5Gnbk6/PKGCS1vpzAKXUQOAw0Av4\nUClVERgC9NNar8vpMxhIUUq11Vpv9kLcflWnTh1/h+A2id1ZWlYagNMV3S1boGJF+xQFiwW++AJu\nuAGWLIE77nDvc+S4B71Sj7U+iTIAmf38k/yMLxhy9BVvTF3oCfyolPpQKXVYKZWslHIMxEqp+kA1\n4OvcNq31SeAHoF1OUxvsRXjePqnA3jx9TGXEiBH+DsFtEruzI5lHAJyu6G7ZAgkJ9iIXoHZt+Pxz\niIiAFi3c+xw57kHPE2NtUDL7+Sf5GV8w5Ogr3ih0GwAPA6lAV2AuMEMpdV/O9mqAxn5VIa/DOdsA\nqgLncwblwvoIEZDSMu1XdGMjLk8p37IF2rZ17nfttfZ5uuPH+zI6YSKeGGuFEMLUvFHoWgCb1nq8\n1vonrfW/gX8DD3nhs1x0794dq9Xq9GrXrh3Lly936rdq1SqsVqvL+x999FHmz5/v1JacnIzVaiU9\nPd2p/bnnnuOVV15xatu7dy9Wq5Xt27c7tc+cOdNlzk1WVhZWq9XlCSiLFy9m8ODBLrH17dtX8jBA\nHpMenkSoJZRK5SoBcPAg7Nv3KCdPuuYxYoSVs2cDMw+z/Hu4m8fixYuxWq3ExcXRvHlzrFYrY8aM\ncdmXH3ltrJVxVPKQPIIzD+gLLM/XtgpwzQMeBebna0vO6Zubx+Kcr+OA5oCVfbxXwL68SGvt0Rfw\nJ/BmvraHgL9y/l4fyAZa5evzDTAt5++dgUtAxQL2PaqQz40HtM1m00aUkpLi7xDcJrE7S1qbpKu/\nVt3x9SefaA1a79nj2c+R4+57NptNY79KGq89PHaW9OWJsbaAfRp6HC0uo55/xSX5GZ8vc7w8rtk0\naK+/4pjk03HUG1d0v8VeuucVR85NElrr3cAhoEvuxpybz64HNuU02YCL+frEAXWA77wQs9+NGzfO\n3yG4TWJ3ln8N3S1bIDbWPi/Xk+S4Bz1PjLVByeznn+RnfMGQo694Y9WFacC3SqmnsN/Vez3wD+DB\nPH2mA88qpf7AflViErAP+ATsN0wopeYDU5VSx4FTwAzgW23CFRcAZs2a5e8Q3CaxO0vLSnNZceG6\n60Apz36OHPegV+qxNliZ/fyT/IwvGHL0FY8XulrrH5VSfwMmA+OB3dinGyzJ02eKUioceAOIAjYA\nt2utz+fZ1Rjs0xeWAuWAr7BPCDElIy8lIrE7O5J5hGqR9nt9tLYXuiNHevxj5LgHOQ+OtUHH7Oef\n5Gd8wZCjr3jjii5a6y+BL4vokwQkXWH7OWBEzksIw0jLTKNVbCsAdu+GY8fsV3SF8DRPjLVCCGFm\nXnkEsBDBLO8c3c05E22k0BVCCCF8TwrdAJF/+RAjkdgvy9bZHM066piju2UL1KsHMTFXfp875LgL\n4R6zn3+Sn/EFQ46+IoVugMjKyvJ3CG6T2C87fuY4l/QlxxXd3BvRvEGOuxDuMfv5J/kZXzDk6CtK\n29dONDylVDxgs9lsxMfH+zscEaRS0lJoPqc56wetp32tRCpVgueeg3xreAuDSk5OJiEhASBBa53s\n73g8TcZRIYLP5XHNhn0pbe+K4wVSGQ8+Gkfliq4QHpSWZX/8b0xEDCkpkJkp83OFEEIIf5FCVwgP\nSsu0F7qxEbFs2WJfO9d+AVAIIYQQviaFboDI/3xrI5HYL0vLSqOMKkNUWBSbN0OzZlChgkc/wkGO\nuxDuMfv5J/kZXzDk6CtS6AaIIUOG+DsEt0nsl6VlplElvAoWZfHqjWggx10Id5n9/JP8jC8YcvQV\nKXQDRFJSkr9DcJvEflnuGrrnzsHPP3u30JXjLoR7zH7+SX7GFww5+ooUugHCyHc4S+yXHck8Qkx4\nDL/8AhcuQJs2Ht29EznuQrjH7Oef5Gd8wZCjr0ihK4QH5V7R3b3b/nWTJv6NRwghhAhmUugK4UFp\nmWnEhseybx+Eh0NUlL8jEkIIIYKXFLoBYv78+f4OwW0S+2W5V3T37YOaNe3Li3mLHHch3GP280/y\nM75gyNFXpNANEMnJxn3IksRup7UmPSudmPAY9u+HWrU8tusCyXEXwj1mP/8kP+MLhhx9RR4BLISH\nHD9znMpTKvPR3R8xfehdNGgA77zj76iEJ8kjgIUQZiOPABZCFMuRzCMAxIRfnroghBBCCP+RQlcI\nD0nLsj/+96ryMRw44P2pC0IIIYS4Mil0hfCQtEx7oavOxHDhghS6QgghhL9JoRsgrFarv0Nwm8Ru\nl5aVhkVZyEyvDHh/6oIcdyHcY/bzT/IzvmDI0Vek0A0Qw4cP93cIbpPY7dIy07iq/FUcOlAG8P4V\nXTnuQrjH7Oef5Gd8wZCjr0ihGyC6du3q7xDcJrHb5V1DNyQEYmM9tusCyXEXwj1mP/8kP+MLhhx9\nRQpdITzkSOYRx4oLNWqARb67hBBCCL+SH8VCeEjuFV1fPCxCCCGEEEWTQjdALF++3N8huE1it0vL\nTHNc0fVFoSvHXQj3mP38k/yMLxhy9BUpdAPE4sWL/R2C2yR2u7SsNGIjYn32sAg57kK4x+znn+Rn\nfMGQo69IoRsgPvjgA3+H4LZgiz0tM439J/c7tWmtSctMo0q476YuBNtxF8JTzH7+SX7GFww5+ooU\nukKUUP+P+9PlnS5k62xH28lzJ7mQfYEIYsjMlDm6QgghRCCQQleIEvj1yK98vftrUo+msvKPlY72\nI5lH7H/JjAF8M3VBCCGEEFcmha4QJTBr8yyqR1bn2mrXMv2H6Y72tCz743/Pn7AXunJFVwghhPA/\nKXQDxODBg/0dgtuCJfbjZ47z7s/v8lCbh3is3WOs2rmKlLQUwD5vFyAzLQaloHp1r4TrJFiOuxCe\nZvbzT/IzvmDI0Vek0A0QRn4KSrDEvmDbAi5cusDQhKH0adGHapHVmPHDDMB+RVehyDh4FbGxULas\ntyK+LFiOuxCeZvbzT/IzvmDI0Vek0A0Q/fv393cIbguG2C9lX2L2ltmOArdsmbI83OZh3vn5HY6f\nOU5aZhqVy1fm4P4Qn01bCIbjLoQ3mP38k/yMLxhy9BUpdIUohhV/rGDX8V0Mbzvc0TYsYRgXsy8y\nL3me46lovnpYhBBCCCGKJoWuEMUwc/NM2tRow/U1r3e0VY2sSv+W/Zm5eSYHTh0gJmcNXVlxQQgh\nhAgMUugGiI0bN/o7BLeZPfbU9FRW7VzFiLYjUEo5bRt1/Sj+OvkXn/3+mc+v6Jr9uAvhLWY//yQ/\n4wuGHH1FCt0AMWXKFH+H4Dazxz5r8yxiwmPo26Kvy7Zrq19Lx7odybqQRXTZGI4d812ha/bjLoS3\nmP38k/yMLxhy9BUpdAPEkiVL/B2C28wc+8lzJ1n400KGJgylXEi5AvuMun4UAGGXYgHfTV0w83EX\nwpvMfv5JfsYXDDn6ihS6ASI8PNzfIbjNzLF/lvoZp8+f5qE2DxXa5864O7mlwS3Uph3guyu6Zj7u\nQniT2c8/yc/4giFHX5FCV4grSD6YTIPoBtSqWHj1WsZShtX3rabmmdsBuRlNCCGECBRS6ApxBdsO\nb+OaatcUq+/+/RAVBRERXg5KCCGEEMUihW6AGDt2rL9DcJtZY9das/XgVq6pWrxC19dr6Jr1uAvh\nbWY//yQ/4wuGHH1FCt0AUadOHX+H4Dazxv7Xyb84fvY411a/tlj72rfPt9MWzHrchfA2s59/kp/x\nBUOOvqK01v6OwSOUUvGAzWazER8f7+9whAl8mvopdy65k7/G/HXFObq52raFVq1g3jwfBCf8Ijk5\nmYSEBIAErXWyv+PxNBlHhQg+l8c1G+D97/s4XiCV8eCjcVSu6ApRiK0Ht3JV+auoWaF4l2nl8b9C\nCCFEYJFCV4hCbDu8jWurX+vyNLSCXLgAhw7JigtCCCFEIJFCN0Bs377d3yG4zayxbzu0rcAb0b74\nAt57z7nt0CHQ2rdXdM163IXwNrOff5Kf8QVDjr4ihW6AGDdunL9DcJsZYz9x9gR/nvjTZWmx33+H\nu++GQYMgOc/Mon377H/6stA143EXwhfMfv5JfsYXDDn6ihS6AWLWrFn+DsFtZox926FtAE6F7sWL\ncCzl3D8AACAASURBVN999mK2ZUsYMsQ+ZQHsa+iCb6cumPG4C+ELZj//JD/jC4YcfUUK3QBh5KVE\nzBj7tkPbCAsJI65KnKPtpZfAZoN334UFC+DXX+GVV+zb9u2D8uUhOtoXUduZ8bgL4QtmP/8kP+ML\nhhx9JcTfAQgRiLYd2sbVsVcTYrF/i2zZAhMnwjPPwPXX2/uMGweTJkHv3pdXXCjGfWtCCCGE8BG5\noitEAbYe2uqYtpCVZZ+ycO218Oyzl/tMmAANGtinMOzdKysuCCGEEIHG64WuUupJpVS2UmpqvvaJ\nSqkDSqkspdRqpVSjfNvLKaVmK6XSlVKnlFJLlVKx3o7XX17J/R24AZkt9nMXz/Fb2m9cW83+RLQn\nnoA9e+xTFkJDL/cLC4O33oLNm+E///H9GrpmO+6idNwda4OR2c8/yc/4giFHX/FqoauUug4YCvyU\nr/0JYHjOtrZAJrBSKVU2T7fpwB3A34GOQA3gY2/G609ZWVn+DsFtZov9t7TfuJh9kWuqXcP338Os\nWfDqq9C0qev727WDkSPtN6r5utA123EX7ivlWBt0zH7+SX7GFww5+ozW2isvIBJIBW4G1gJT82w7\nAIzJ83VF4AzQJ8/X54C/5ekTB2QDbQv5vHhA22w2LURpvJX8llZJSp86d0pPnKh1dLTWly4V3v/0\naa1vvFHrzz7zXYzCP2w2mwY0EK+9NHaW9FWasbaAfck4KkSQuTyu2bR9RXjvvuKY5NNx1JtXdGcD\nn2mt1+RtVErVB6oBX+e2aa1PAj8A7XKa2mC/US5vn1Rgb54+QnjF1kNbaXJVEyLLRpKcDPHxYLnC\nd0pEBGzcCD16+C5GIfIozVgrhBCm5pVVF5RS/YBrsBes+VXDXskfztd+OGcbQFXgfM6gXFgfIbxi\n26FtjhvRkpOhb18/ByREITww1gohhKl5/IquUqoW9vm192itL3h6/0Xp3r07VqvV6dWuXTuWL1/u\n1G/VqlVYrVaX9z/66KPMnz/fqS05ORmr1Up6erpT+3PPPecyYXzv3r1YrVaXx/fNnDmTsWPHOrVl\nZWVhtVrZuHGj074XL17M4MGDXWLr27dvQOaR23fjxo1O7UbII+8+srKy6GntyY/f/8g11a4hPd2+\nmsLZs4GZR3p6epHnVV6B9O/x3XffObUHYh6LFy/GarUSFxdH8+bNsVqtjBkzxmVf/uLNsdao42he\nVzpP3nnnHVPkUdi/R/7YjJpHfrl55I3ZyHnklT+P3D99nQf0BZbna1sFuOYBjwLz87Ul5/TN/fdY\nnPN1HNAcsLKP9wrYlxd5ei4EcCdwCTgPXMh5Zedpa5Dzdat87/sGmJbz9845/Svm6/MnMKqQzzX0\n3LKePXv6OwS3mSn2P47+oUlCf7XjK71ypX0+0e+/+ym4IpjpuBtFIM3R9cRYW8A+DT2OFpdRz7/i\nkvyMz5c5yhzdkvsvcDX2X6e1znn9CLwHtNZa7wIOAV1y36CUqghcD2zKabIBF/P1iQPqAM6Xgkwi\nKSnJ3yG4zUyx5330b3IyVKgADRv6IbBiMNNxF27xxFgblMx+/kl+xhcMOfqKx+foaq0zgd/ytiml\nMoGjWuuUnKbpwLNKqT+wX6WdBOwDPsnZx0ml1HxgqlLqOHAKmAF8q7Xe7OmYA0F8fLy/Q3CbmWLf\ndmgb1SKrUTWyKsnJ9odEXOlGNH8y03EXJeeJsTZYmf38k/yMLxhy9BVfPQJYO32h9RSlVDjwBhAF\nbABu11qfz9NtDPZfwS0FygFfYZ8QIoTX5H0iWnIy9Ozp54CEKBl3xlohhDAtnxS6WuubC2hLApKu\n8J5zwIiclxA+se3QNga2HsiJE7BzJyQk+DsiIYrPnbFWCCHMLEB/KRt88t+BaSRmif23tN/Yf2o/\nbWu2ZetWe1sg//bILMddCF8z+/kn+RlfMOToK1LoBojk5GR/h+A2s8Q+d8tcYiNi6d64O8nJUL48\nxMX5MbgimOW4C+FrZj//JD/jC4YcfUVprYvuZQBKqXjAZrPZZBK3KLHT509Tc2pNhl83nBe7vMg9\n98Du3bApqO9NF/klJyeTYJ/PkqC1Nt1PIhlHhQg+l8c1G/YVBr0rjhdIZTz4aByVK7pCAIt+WcTp\n86cZmjAUAJtN5ucKIYQQRieFrgh6Wmtmb5lNjyY9qBtVl1On4PffA3t+rhBCCCGKJoWuCHrf7fuO\nnw//zMNtHgbgp5/sz2+RQlcIIYQwNil0A0RBz8M2CqPHPvfHuTSIbkDXhl0B+/q55cpB8+Z+Dq4I\nRj/uQviL2c8/yc/4giFHX5FCN0AMHz7c3yG4zcix3/uPe/nwfx/ycJuHsSj7t4PNBq1aQWion4Mr\ngpGPu5FjF8Zn9vNP8jO+YMjRV6TQDRBdu3b1dwhuM3LsuyvvRqEYfM1gR1tysjGmLRj5uBs5dmF8\nZj//JD/jC4YcfUUKXRG0Lv1/e3ceXkV1/3H8fcK+iIpsVUAFFKUuyCp1X1GUoEgratUCLS74q/io\nWLeKdFGwdUFwaaFaRCOKgktRkFIwAUFJEGQVBAnKogFkSyAhOb8/5hKSEJZcbmbmzP28nuc+D5k7\n997Pd3Jy+GYyS1EhL2e+zPWnXc8xtY8BIDcXFi92o9EVERGRA/PlFsAiYTT5m8ms+mkVadelFS/7\n6isoKlKjKyIiEgXaoxsSEydODDpC3FzNPvKLkZy4/kQ6HdepeFlmJlStCqefHmCwQ+Tqdge3s4v7\noj7+VJ/7kqFGv6jRDYm0tLSDrxRSLmZfvnE5k5ZPosE3DTDGFC/PyoLTTvOuuhB2Lm73PVzOLu6L\n+vhTfe5Lhhr9okY3JMaNGxd0hLi5mH3kFyM5ptYxzPjPjFLLXTkRDdzc7nu4nF3cF/Xxp/rclww1\n+kWNriSdbbu28a95/6J/+/7UqlarePmmTbBwoTuNroiIiByYGl1JOmPmjyG3ILf4Tmh7DBkCNWtC\nr14BBRMREZGEUqMrkVNQWMCwmcNYt23dPs8V2SKe//x5ep7ak2ZHNitevmwZjBwJDz0EjRv7mVZE\nREQqixrdkOjTp8/BVwqpsGWfuWYmD0x9gO5p3cktyC313CfffMKyjcv4feffA3uzDxoExx0HAwf6\nHjduYdvuFeFydnFf1Mef6nNfMtToFzW6IeHyXVDClj0jO4M61eqwNGcpt0y4hSJbVPzc8M+Hc1aT\nszin2TmAl33aNHj/fRg61Dt0wRVh2+4V4XJ2cV/Ux5/qc18y1OgXY60NOkNCGGPaAZmZmZm009lE\nSa3r2K5UTalK/3b9uXbctTx47oP85ZK/sHzjck4ecTKv9HiF37T9DQCFhdChg9fgzpoFJa40JrKP\nrKws2rdvD9DeWpsVdJ5E0zwqknz2zmuZQOX/3LfmzyzjUfBpHtWd0SRSdhftZtaaWTx83sP0OKUH\nQy8dyqCpg2jdoDWZazNpULsBvU/rXbz+mDHw5Zfw2WdqckVERKJGja5Eyvz189mev53zmp8HwH2/\nuI+lOUv53Qe/o1pKNe7ufDc1q3rHJ2zf7p18dsMNcPbZQaYWERGRyqBjdEMiIyMj6AhxC1P29Ox0\nalSpQYdjOwBgjOHFq1+kS9Mu7CrcxR0d915S7KmnYOPGDJ54Iqi0hydM272iXM4u7ov6+FN97kuG\nGv2iRjckhg0bFnSEuIUpe3p2Op2bdqZG1b338K1epTof3vghWf2zaFqvKQBbtsCzz0Lz5sM4/vig\n0h6eMG33inI5u7gv6uNP9bkvGWr0ixrdkHjzzTeDjhC3sGS31pK+Op1zm527z3N1q9fl9ManF3/9\n0kuwcydMnhyO7PEIy3aPh8vZxX1RH3+qz33JUKNf1OiGRO3atYOOELewZP9649f8mPsj5x1/3gHX\n27kTnnkGbr0VWrYMR/Z4hGW7x8Pl7OK+qI8/1ee+ZKjRL2p0JTIysjNIMSn8otkvDrjemDHwww9w\n330+BRMREZFAqNGVyEjPTufMxmdSr0a9/a5TWOidhNazJ5x8so/hRERExHdqdEPi/vvvDzpC3MKS\nPT07vfiyYvszYQKsWAEPPOB9HZbs8VB2kfhEffypPvclQ41+UaMbEs2bNw86QtzCkH3ttrWs3Lzy\ngMfnWgtPPgkXXwwdO3rLwpA9XsouEp+ojz/V575kqNEvugWwRMK4hePo/U5v1t+7nsZ1G5e7zn//\nC5deCpMng24jLvHQLYBFJGqifgtg7dGVSEjPTuek+iftt8kFGDoUzjoLLrvMx2AiIiISGN0CWCIh\nPTudc5vvvX7ulCmwdi3s2OE9cnLgk08gLQ2MCTCoiIiI+EZ7dENi6dKlQUeIW9DZf9r5E19t+Kr4\nRLRPP4WuXaFPH7jnHnjiCXjjDejRA3r1Kv3aoLMfDmUXiU/Ux5/qc18y1OgXNbohMWjQoKAjxC3o\n7LPWzMJii09EGz4cTj0Vdu2C/HzYvBm++w4mToSqZf6GEXT2w6HsIvGJ+vhTfe5Lhhr9okMXQmLE\niBFBR4hb0NnTV6fTpG4TWh7dkuxs7xJiI0dC9eoHf23Q2Q+HsovEJ+rjT/W5Lxlq9Iv26IaEy5cS\nCTL7oh8W8fbitzmv+XkYY3jhBTjiCPj1rw/t9druwXA5u7gv6uNP9bkvGWr0ixpdcVJuQS4PTn2Q\nti+3pWpKVR45/xHy8uCf/4R+/aBu3aATioiISNB06II456PlHzFg0gDWblvLH8//I4POGUSNqjUY\nPdo7HnfAgKATioiISBhoj25IDB06NOgIcfMz+3Ozn6PbG91oWb8lX93xFY9e8Cg1qtbAWu8ktKuv\nhhYtDv39tN2D4XJ2cV/Ux5/qc18y1OgX7dENidzc3KAjxM2v7HkFefwl/S/0bduXUamjMCUuiPvp\np7BgAfz97xV7T233YLicXdwX9fGn+tyXDDX6RbcAFme8NPclBkwawNd3fU3L+i1LPderFyxeDIsW\n6YYQUnl0C2ARiZqo3wJYe3TFCYVFhfxt1t/o1abXPk3unkuKjRihJldERET2UqMrTnhnyTt8s/kb\nxvUat89zey4pdvPNAQQTERGR0NLJaCGRk5MTdIS4VXZ2ay3DZg7jkhMvof2x7Us9t2kTvPQS/Pa3\n8V1STNs9GC5nF/dFffypPvclQ41+UaMbEn379g06QtwqO/u0VdPIXJfJA+c8sM9zTz0FBQVw//3x\nvbe2ezBczi7ui/r4U33uS4Ya/aJGNyQGDx4cdIS4VXb2YbOGcVaTs7i0xaWllq9f711SbOBAaNw4\nvvfWdg+Gy9nFfVEff6rPfclQo190jG5IuHyGc2Vmn7duHlO+mULadWmlLicG8Ne/QvXqcN998b+/\ntnswXM4u7ov6+FN97kuGGv2iPboSak/NeooTjzqRXm16lVq+erV3bO6gQXD00QGFExERkVBToyuh\ntfqn1YxbNI57u9xL1ZTSf3x4/HGvwf397wMKJyIiIqGnRjckRo8eHXSEuFVW9rcXv031KtW5te2t\npZYvWwb//jc8/DDUqXN4n6HtHgyXs4v7oj7+VJ/7kqFGvyS80TXGPGiM+dwYs9UYs8EYM8EYc3I5\n6w0xxqw1xuQaYz4xxrQq83wNY8xIY0yOMWabMWa8MaZRovOGRVaWuzdZqqzs7yx5hytaXUHd6qWv\nG/bHP8Jxx8Fttx3+Z2i7B8Pl7GGRqLk2GUV9/Kk+9yVDjX5J+C2AjTGTgDRgLt7Jbk8ApwGnWmvz\nYus8ADwA3AJ8C/wZOD22Tn5snReBK4Fbga3ASKDQWnvefj5Xt66MkO+3fk/TZ5oy5pox3Hzm3jtB\nzJsH7drBqFHQr1+AASUphekWwImaa8u8p+ZRkSSjWwBXkLW2W8mvjTG/AX4A2gMZscV3A3+y1n4Y\nW+cWYANwDfCWMaYe0Bfoba2dEVunD7DEGNPJWvt5onNLuExYOoGqKVXp3rp7qeVDhkCrVnDrrft5\noUiSSMRc61tYEZGA+HGM7lGABTYBGGNOBJoA/92zgrV2KzAH6BJb1AGvCS+5zjIgu8Q6EmHvLnmX\nS068hKNqHlW8bNEimDgRHnoIqurCeCJlxTPXiohEWqU2usa78OmzQIa1dnFscRO8yXhDmdU3xJ4D\naAzkxybl/a0jEfXjjh+ZsXoG1516XanlTz4JzZrBTTcFFEwkpA5jrhURibTK3qP7AtAG6F3Jn1Os\nW7dupKamlnp06dKFiRMnllpvypQppKam7vP6AQMG7HO2Y1ZWFqmpqfvce/qxxx5j6NChpZZlZ2eT\nmprK0qVLSy1//vnnub/MfWpzc3NJTU0lIyOjVJa0tDT69OmzT7brr78+lHXs2eYZGRmllsdbx/vL\n3gegxyk9iutYuRLS0rxb/S5cmLg6Sm67kt+PRNRRUmV8P1JTUw86rsJax0UXXVRqeRjrSEtLIzU1\nldatW9OmTRtSU1O555579nmvkEjoXOvqPFrSgcZJx44dI1HH/r4fZWtxtY6y9tRR8jmX6yipbB17\nXud3HXA9MLHMsinAvnXAAKDs1SGyYuvu+X6kxb5ujTdFpfIdY8t5r0pkra2UBzACWA00L7P8RKAI\nOKPM8unAM7F/XwQUAvXKrPMtcPd+Pq8dYDMzM62LJk+eHHSEuCU6+5Vjr7QXvHJBqWW33WZto0bW\n5uYm9KO03QPiavbMzEyLt5e0na2kubOij8OZa8t5L6fn0UPl6vg7VKrPfX7WuHdey7RgK/3Rmj/5\nOo9Wyh5dY8wIoAdwkbU2u+Rz1tpVwHrgkhLr1wM6A7NiizKB3WXWaQ00Bz6rjMxBu/zyy4OOELdE\nZt+ycwtTV06l56k9i5etXQuvvAL33AO1aiXsowBt96C4nD1MEjDXJqWojz/V575kqNEvCT+lxxjz\nAnAD3r7qHcaYxrGntlhrd8b+/SzwiDFmBd5e2j8B3wHvgXfChDFmNPC0MWYzsA0YDsy0uuJCpH34\n9YcUFBVw7SnXFi97+mmvwb3jjgCDiYRMIuZaEZGoq4xz12/H2yU9vczyPsAYAGvtMGNMbeBlvDOF\n04ErbenrOt6Dd/jCeKAG8DHeASESYe8ufZdOx3Wi2ZHNANi4EV56CQYOhCOPDDicSLgkaq4VEYms\nhB+6YK1NsdZWKecxpsx6g621x1pra1tru1prV5R5fpe19v+stQ2stUdYa39prf0h0XnDouxB7C5J\nVPYd+Tv4aPlHpa628Pzz3lE9Awcm5CP2oe0eDJezh0Wi5tpkFPXxp/rclww1+sWP6+jKIUhLSws6\nQtwSlX3yN5PJ251XfHzutm0wfDj07w8NGiTkI/ah7R4Ml7OL+6I+/lSf+5KhRr+o0Q2JcePGBR0h\nbonKPn7xeM5ofAat6rcC4G9/g9xcuPfehLx9ubTdg+FydnFf1Mef6nNfMtToFzW6EqjcglxemfcK\nnUd1Jm1hGjed7t0NYt06r9EdOBCaNg04pIiIiDhJN1KVQGRvyeaZz57h1fmvsmXnFrq26srE6yfS\nvXV3AB57zLvSwoMPBhxUREREnKVGV3w35Zsp9B7fmyopVejfrj+3dbiNFke3KH5+8WIYPdq7rJiu\ntCAiIiLx0qELIVH+rfjccKjZrbUMzRjKla9fydlNz+bru75m6GVDSzW5AA88ACec4M91c5Nhu4eR\ny9nFfVEff6rPfclQo1+0RzckXL4LyqFk356/nb7v9eXtxW/z0LkPMeSiIVRJqbLPetOnw4cfwrhx\nUL16JYQtI+rbPaxczi7ui/r4U33uS4Ya/WKsd39z5xlj2gGZmZmZtGvXLug4UsKKTSvoOa4nKzev\n5N/X/Jvr2lxX7npFRdC5M6SkwOzZYIzPQUUOIisri/bt2wO0t9ZmBZ0n0TSPiiSfvfNaJlD5P/et\n+TPLeBR8mke1R1cq1btL3qXPe31oXKcxc347h583+vl+133rLZg7F2bMUJMrIiIih0/H6EqlKCgs\n4N7J93LdW9dxecvLmdt/7gGb3JUr4b77IDUVzj/fx6AiIiISWWp0QyIjIyPoCHErm/37rd9z0b8v\nYvjnw3mm6zO81est6tWot9/XL1oE554LtWvDyJGVnba0KG13l7icXdwX9fGn+tyXDDX6RY1uSAwb\nNizoCHErmX1H/g46jerE6i2rmfGbGQw8eyDmAMchzJ0LF1wADRtCerr/N4eIynZ3jcvZxX1RH3+q\nz33JUKNfdIxuSLz55ptBR4hbyeyvf/U667atY/n/Ladl/ZYHfN2nn8LVV8PPfw6TJsHRR1d20n1F\nZbu7xuXs4r6ojz/V575kqNEv2qMbErVr1w46Qtz2ZLfWMvKLkXRv3f2gTe6UKdC1K3TsCJ98EkyT\nC9HY7i5yObu4L+rjT/W5Lxlq9IsaXUmYjOwMFmxYwICOAw643mefwTXXwMUXw3/+A3Xr+hRQRERE\nkooaXUmYEV+M4ORjTubSFpfud51Fi+Cqq6B9exg/HmrW9DGgiIiIJBU1uiFx//33Bx0hbvfffz9r\nt63l3SXvMqDjAFJM+cMqO9s7XKFpU/jgA6hVy+eg5XB9u7vK5ezivqiPP9XnvmSo0S9qdEOiefPm\nQUeIW/PmzflH5j+oUaUGt555a7nr5OTA5ZdDtWoweTIcdZTPIffD9e3uKpezi/uiPv5Un/uSoUa/\n6BbActjyC/M5/tnjuab1Nbx49Yv7PL9mDfTqBatWwcyZcNJJAYQUSQDdAlhEoibqtwDWHl05bBOW\nTGD99vUM6LT3JLTCQu9Es9RUOOEEWL4cPv5YTa6IiIj4R42uHLYRX4zgwhMu5LRGpwHw3HPQooV3\njdzvvoMXX4TVq0E7iERERMRPanRDYunSpUFHiMv89fPJyMzgro53ATBjBgwcCOedB3PmQGYm9O8P\nRxwRcND9cHW7g7KLxCvq40/1uS8ZavSLGt2QGDRoUNARKmzrrq08MPUBav6vJj1O6QHA4MHQti28\n9hp06gQHuPtvKLi43fdQdpH4RH38qT73JUONftEtgENixIgRQUeokIzsDG6ecDMbczcyYsQIqqZU\nZfp0mD4dJkwIf4O7h2vbvSRlF4lP1Mef6nNfMtToF+3RDQlXLiVSUFjAI9Me4YJXL6BpvaYsuGMB\n/S7qB8Djj3t7c3v0CDhkBbiy3cuj7CLxifr4U33uS4Ya/aI9unLIVv+0ml++/UvmrZ/HkAuH8Idz\n/0CVlCoAxXtzJ050Z2+uiIiIRJsaXTkka7et5eIxF1Nki5jVdxYdj+tY6vk9x+ampgaTT0RERKQs\nHboQEkOHDg06wn7l5OZw2WuXkV+Yz/9u/d8+Te5ttw1lxgyv2XVtb26Yt/vBKLtIfKI+/lSf+5Kh\nRr9oj25I5ObmBh2hXFt2buGKsVfw444fSe+TzglHnbDPOpMn53LWWW7uzQ3rdj8Uyi4Sn6iPP9Xn\nvmSo0S+6BbDsV25BLl3HdmXhDwuZfut0zmxy5j7rTJ8OF10E773nZqMrUhG6BbCIRI1uASxJqaCw\ngJ7jejJv3Tw+uumjfZrcoiLvWrm9e3t3POvePaCgIiIiIvuhRlfKdc/ke5i2ahrv9X6Ps5ueXeq5\nrCzvzme33AIXXADvv+/esbkiIiISfWp0QyInJyfoCMVGZY1i5BcjGdFtBJe0uKR4eU4O3H47dOgA\nW7bAtGkwbhzUqBGe7BUVpu1eUcouEp+ojz/V575kqNEvanRDom/fvkFHAGDWmlnc+Z87ub397fRv\n3x+AvDwYOhRatoS0NHjmGZg3zzs2F8KTPR7KHgyXs4v7oj7+VJ/7kqFGv+iqCyExePDgoCPw3dbv\n6DmuJ52bdua5K5+jqAjeeAMeegjWrYM77oBHH4WGDUu/LgzZ46XswXA5u7gv6uNP9bkvGWr0i/bo\nhkTQZzjv3L2TnuN6Ur1Kdcb/cjzZq6rTuTPcfDN07AiLFsHw4fs2uRB89sOh7MFwObu4L+rjT/W5\nLxlq9Iv26CaZbbu28d6y95jz3Rx2FOwgtyCX3IJcvv3pW5ZvWk5Gnwyy0htz443QqBGkp8O55wad\nWkRERKTi1OgmgV27d/HRio9IW5jGB8s+IG93Hm0atuHIGkdSu1ptalerTZuGbRh66TAmv9qeRx6B\nq66CsWPhyCODTi8iIiISHx26EBKjR4+ulPcdt3Acxz59LNeOu5ZlOcsYfOFgvr37WxbduYhZ/WYx\n9ZapvH/D+4y64k1GP3gFDz8Mjzzi3QDiUJvcysruB2UPhsvZxX1RH3+qz33JUKNf1OiGRFZWYm8O\nsnP3Tu78z530fqc3l7W4jEV3LmJOny857ttB3HT18RhDqccRR8DkyTBhAgwZAikVGBmJzu4nZQ+G\ny9nFfVEff6rPfclQo190C+AIWrFpBb96+1cs/nExw68czqVH/45//MMwerR3LdyLL4aePaFGjb2v\nsda7XFirVsHlFgk73QJYRKIm6rcA1jG6EWGtZc3WNXy84mPum3Ifjes25r3usxn3XFvu+Le3x7ZP\nH++GD61bB51WREREpPKp0XVUQWEBX6z9gvTV6cz+fjazv5vN+u3rAeh5cm+affky15xdjyOOgGef\nhX79oHbtgEOLiIiI+EiNbkhtztvM1l1bSy37aedPTP92OlNXTWXGtzPYlr+NutXr0um4TtxwSh+O\n2n42Py3qzKsDGpOfD4MGwb33Qr16ARUhIiIiEiCdjBYSqampACzfuJx+7/Wj0d8accJzJ5R6tH25\nLYOmDmL7rlxubP4HBtaZQ7cFm1k95L88c/Vfeax3KqOfa8z118OKFfD44/40uXuyu0jZg+FydnFf\n1Mef6nNfMtToF+3RDYmrbrqKG965gbcWvUWj2o3o3+IJ6ue3JS8PduRCXi5s3VST7NntmZlVi+kF\nULMmtGsHV1/t3b2sY0fvZLKKXDEhEe666y5/PzCBlD0YLmcX90V9/Kk+9yVDjX5Ro1uJsrdk8+qX\nr5JfmE/d6nWLHykmhe+3fs+arWtYs3UN2VuyWbBhAcdUac6Z349gxdt9eGFzzeL3qVPHO5ns173I\n/QAADEpJREFUmGO8xrbvLdC5M5xxBlSrFmCBMZdffnnQEeKm7MFwObu4L+rjT/W5Lxlq9Isa3Uqw\nYMMCnpr1FGlfpVG7Wh3qVjmK7QXbySvczm6bD0CNwvpU39WUlO3NKNr0C1hyLxsX3kCrDtW4/x7o\n1g1atoS6daGqvksiIiIiFaYWqoJyC3IZv3g8/8j8B1nrsmhW73haHt2ClvVbcPxRxzNp6X/535qP\nqVvYnAbz/86Gj/uxLb9u8etr1S2gYeNCftagJo0a4T1OhFOuhCuu8L4WERERkcMX+UZ3Y+5Gvvrh\nK77a8BULNizg601fs3P3TgoKCygoLCC/sID6terTtdXlXNnqSjoc24EqKVVKvUd+YT4L1i/k79Ne\n4b1Vr5Fnt1A1+1J2L32cr49Yy9dHr4T60+HolbDxJJg5lp/l/4qLLqjG+f+CFi2gcWPvUadONWDf\n4w0mTpxIo0bX+LNREmzixIlcc42y+03ZReIT9fGn+tyXDDX6JXJ3Rhv14WjWVN3MjJWzWbBxDpsK\n13jPF1WnxpY2FP5wCkU761BUUA1bWA0Kq8GR2ZhWn2BrbKFawTE0y7+cKrYWm+xKtlddya4a30FK\nEWxrQsqCPnRI6Ue3Li054wwoKIDcXO+xY4elaVPDhRfCz35WsfxdunThs88+S/h28YOyB0PZ/ac7\no0WDq+PvUKk+9/lZo+6MFjBjzADgPqAJMB/4P2vtF/tb/7fv94MGtWFtB/i+N0dsb0ez6mfQ6uiT\nOL5ZNY49A2rVgurV9z527YLv1u7my5zZLM7/mO9rTcFgqFd4Iift7kIT24IT6rXil1ecw3l/qkat\nWvtNG3edDRs2jPu1QVP2YCi7HKqKzqNRF/Xxp/rclww1+iXUja4x5nrg70B/4HPgHmCyMeZka21O\nea+5seYbXNfxl5zcqiotWlTkbmBVgXNjjz8ffngRkRCIZx71U3Z2Njk5/sbIy8vz9fMksfLy8sjK\n8vcPKg0aNKB58+a+fqYkRqgbXbwJ+WVr7RgAY8ztwFVAX2BYeS+499bWtGsX9rJERHxT4XnUL9nZ\n2bRufSo7d+b6+rkpKVXIzs5W4+Kg7Oxspk37355DiHxTs2Ztli1bojHjoNB2hMaYakB74K97lllr\nrTFmKtAlsGAiIo4I+zyak5MTa3LHAqf69KlLKCr6NTk5OWpaHJSTk0NRUSF+j5mdOzVmXBXaRhdo\nAFQBNpRZvgFoXc76NQGWLFlSybEqx+eff+77n2ISRdmDoez+KzG/1DzQeiES1zz67rvvMnfu3EqO\n5u2d86yq9M/ay/usSZMm+fb/RUpKCkVFRb58FsDMmTN5/fXXffs8P+tbtWrPWIn2mPHze7h3m04C\nKr++nczf809f5tHQXnXBGPMz4Hugi7V2TonlQ4HzrbVdyqx/I+DfT7aIJLObrLVvBB3iYDSPikiI\n+TKPhnmPbg5QCDQus7wxsL6c9ScDNwHfAjsrNZmIJKuawAl4840LNI+KSNj4Oo+Gdo8ugDFmNjDH\nWnt37GsDZAPDrbVPBRpORMQBmkdFJJmFeY8uwNPAq8aYTPZeFqc28GqQoUREHKJ5VESSVqgbXWvt\nW8aYBsAQvD+1fQl0tdb+GGwyERE3aB4VkWQW6kMXRERERETilRJ0ABERERGRyqBGV0REREQiKRKN\nrjFmgDFmlTEmzxgz2xjTMehM5THGnGeMed8Y870xpsgYk1rOOkOMMWuNMbnGmE+MMa2CyFom04PG\nmM+NMVuNMRuMMROMMSeXs14Ys99ujJlvjNkSe8wyxlxRZp3Q5S6PMeYPsXHzdJnloctvjHkslrXk\nY3GZdUKXew9jzLHGmNeMMTmxfPONMe3KrBPa/PFyZS4tKVHzkzGmhjFmZOx7vs0YM94Y08i/Sg5N\nvPNA2OtLxM9cWGs0xqQYY/5kjFkZy77CGPNIOes5U59JQD9zKPUYY442xrwe+/97szFmlDGmToXC\nWmudfgDX413v8RbgFOBlYBPQIOhs5WS9Au+EkB5417ZMLfP8A7HsVwOnAROBb4DqAeeeBNyMd7/F\n04EP8a6zWcuB7FfFtntLoBXwZ2AXcGqYc5dTR0dgJTAPeNqB7f4YsABoCDSKPeqHPXcs21F4t0Ia\nhXf73OOBS4ETXch/GHU7M5eWyZ2Q+Ql4Mfa6C4CzgFlAetD1lak17nkgzPUl6mcurDUCDwE/4P1f\n1BzoCWwF7nK1PhLQzxxKPcBHQBbQAfgF8DUwtkJZgx4ACdjYs4HnSnxtgO+AQUFnO0juonIGxlrg\nnhJf1wPygF8FnbdMzgax/Oe6lj2WbSPQx5XcQF1gGXAx8D9K/wcXyvx4jW7WAZ4PZe5YlieBGQdZ\nJ7T5D6NuJ+fScuqo8PwU+3oXcG2JdVrH3qdT0DXF8sQ9D4S9vkT8zIW5RuAD4J9llo0HxkSkvgr3\nM4dSD94vr0XAWSXW6QrsBpocaj6nD10wxlTD++3vv3uWWW9LTAW67O91YWSMORFoQulatgJzCF8t\nRwEW77c1Z7LH/nzUG+8aorNcyQ2MBD6w1k4rudCB/CfF/qz1jTFmrDGmGTiRuzsw1xjzVuxP4VnG\nmN/uedKB/BUWpbmU+OanDniX2yy5zjK8G2uEpf7DmQfCXl8ifubCXOMs4BJjzEkAxpgzgXPw/hoR\nhfpKSWA9ZwObrbXzSrz9VLyf786HmifU19E9BA2AKsCGMss34P1m4JImeN+88mpp4n+c8hljDPAs\nkGGt3XPMZaizG2NOAz7Du+3gNrzfIJcZY7oQ4twAsca8Ld6kUFaYt/ts4Dd4e6B+BgwGPo19L8Kc\nG6AFcAfwd+AvQCdguDFml7X2NcKfPx6RmEsPY35qDOTH/jPe3zqBScA8EOr6SMzPXJhrfBJvD+ZS\nY0wh3vlRD1tr34w973p9ZSWqniZ4h3wUs9YWGmM2UYGaXW90xX8vAG3wfht1xVLgTOBIoBcwxhhz\nfrCRDs4Y0xTvP+1LrbUFQeepCGttyXuYLzTGfA6sBn6F9/0IsxTgc2vto7Gv58ca9NuB14KLJYfA\nxfnpgFyeByog6j9z1wM3Ar2BxXi/tDxnjFkba+SlEjl96AKQg3cQdOMyyxsD6/2Pc1jW4x0TF9pa\njDEjgG7AhdbadSWeCnV2a+1ua+1Ka+08a+3DwHzgbkKeG+9PyQ2BLGNMgTGmAO+g/buNMfl4v/mG\nOX8xa+0WvJMIWhH+7b4OWFJm2RK8k0gg/Pnj4fxcepjz03qgujGm3gHWCUoi5oEw1weJ+ZkLc43D\ngCettW9baxdZa18HngEejD3ven1lJaqe9XgnMhczxlQB6lOBmp1udGO/3WYCl+xZFvvT1SV4x8Q4\nw1q7Cu8bV7KWenjHoQReS+w/kR7ARdba7JLPhT17OVKAGg7knop3FnlbvD3SZwJzgbHAmdbalYQ7\nfzFjTF28JnetA9t9Jvv+ub413h5pF8f7Qbk+lyZgfsrEO8Gl5Dqt8Rqtzyo1/MElYh4Ic32QmJ+5\nMNdYG+8XyZKKiPVgEaivlATW8xlwlDHmrBJvfwleEz2nIoGcfuD9KTSX0pfE2Qg0DDpbOVnr4E1S\nbfEG+cDY181izw+KZe+ON7FNBJYT/KWiXgA2A+fh/ba151GzxDphzf7XWO7j8S5x8kTsh+viMOc+\nQD1lz7YOZX7gKeD82Hb/BfAJ3p6nY8KcO5atA97ZwA/iXZbuRrxju3uHfbsfZt3OzKVlcidkfoq9\nzyrgQry9qDMJwaWp9lNzheeBMNeXqJ+5sNYIvIJ3klW32Jx4Ld6xp391tT4S0M8cSj14J+zNxbu0\n3jl45328VqGsQQ+ABG3wO/GuxZaH9xtAh6Az7SfnBbEBUVjm8a8S6wzGuyxHLjAZaBWC3OVlLgRu\nKbNeGLOPwrvuZB7eb5hTiDW5Yc59gHqmUeI/uLDmB9LwLk2VF5vg36DENTHDmrtEtm541wHOBRYB\nfctZJ7T5D6NuJ+bSMpkTMj8BNYDn8Q7j2Aa8DTQKur791FzheSDs9SXiZy6sNeI1hU/jNXU78Bq+\nx4GqrtZHAvqZQ6kH7yoqY4EteL/Q/hOoXZGsJvZGIiIiIiKR4vQxuiIiIiIi+6NGV0REREQiSY2u\niIiIiESSGl0RERERiSQ1uiIiIiISSWp0RURERCSS1OiKiIiISCSp0RURERGRSFKjKyIiIiKRpEZX\nRERERCJJja6IiIiIRNL/A1KbNNUUAUAhAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe3d04edda0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You Win! You may stop training now via KeyboardInterrupt.\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-104-85e9045108b8>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m     \u001b[0;31m#generate new sessions\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m     \u001b[0msessions\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mgenerate_session\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0m_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn_sessions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      8\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      9\u001b[0m     \u001b[0mbatch_states\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mbatch_actions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mbatch_rewards\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmap\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0msessions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-104-85e9045108b8>\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m     \u001b[0;31m#generate new sessions\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m     \u001b[0msessions\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mgenerate_session\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0m_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn_sessions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      8\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      9\u001b[0m     \u001b[0mbatch_states\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mbatch_actions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mbatch_rewards\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmap\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0msessions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-103-5e00e4320706>\u001b[0m in \u001b[0;36mgenerate_session\u001b[0;34m(t_max)\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m         \u001b[0;31m#predict array of action probabilities\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m         \u001b[0mprobs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0magent\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict_proba\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     12\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m         \u001b[0ma\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrandom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchoice\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mprobs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/main/anaconda2/envs/python3/lib/python3.5/site-packages/sklearn/neural_network/multilayer_perceptron.py\u001b[0m in \u001b[0;36mpredict_proba\u001b[0;34m(self, X)\u001b[0m\n\u001b[1;32m   1048\u001b[0m         \"\"\"\n\u001b[1;32m   1049\u001b[0m         \u001b[0mcheck_is_fitted\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"coefs_\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1050\u001b[0;31m         \u001b[0my_pred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_predict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1051\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1052\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mn_outputs_\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/main/anaconda2/envs/python3/lib/python3.5/site-packages/sklearn/neural_network/multilayer_perceptron.py\u001b[0m in \u001b[0;36m_predict\u001b[0;34m(self, X)\u001b[0m\n\u001b[1;32m    658\u001b[0m             \u001b[0mThe\u001b[0m \u001b[0mdecision\u001b[0m \u001b[0mfunction\u001b[0m \u001b[0mof\u001b[0m \u001b[0mthe\u001b[0m \u001b[0msamples\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0meach\u001b[0m \u001b[0;32mclass\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    659\u001b[0m         \"\"\"\n\u001b[0;32m--> 660\u001b[0;31m         \u001b[0mX\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcheck_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maccept_sparse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'csr'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'csc'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'coo'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    661\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    662\u001b[0m         \u001b[0;31m# Make sure self.hidden_layer_sizes is a list\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/main/anaconda2/envs/python3/lib/python3.5/site-packages/sklearn/utils/validation.py\u001b[0m in \u001b[0;36mcheck_array\u001b[0;34m(array, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, warn_on_dtype, estimator)\u001b[0m\n\u001b[1;32m    397\u001b[0m     \u001b[0mdtype_numeric\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msix\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstring_types\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mdtype\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"numeric\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    398\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 399\u001b[0;31m     \u001b[0mdtype_orig\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"dtype\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    400\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdtype_orig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'kind'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    401\u001b[0m         \u001b[0;31m# not a data type (e.g. a column named dtype in a pandas DataFrame)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "n_sessions = 100\n",
    "percentile = 70\n",
    "log = []\n",
    "\n",
    "for i in range(100):\n",
    "    #generate new sessions\n",
    "    sessions = [generate_session(env) for _ in range(n_sessions)]\n",
    "    \n",
    "    if i %10==0:\n",
    "        \n",
    "\n",
    "    batch_states,batch_actions,batch_rewards = map(np.array,zip(*sessions))\n",
    "\n",
    "    elite_states, elite_actions = select_elites( batch_states,batch_actions,batch_rewards, percentile)\n",
    "    \n",
    "    agent.fit(elite_states,elite_actions)\n",
    "\n",
    "    show_progress(batch_rewards,log,reward_range=[0,np.max(batch_rewards)])\n",
    "    \n",
    "    if np.mean(batch_rewards)> 190:\n",
    "        print(\"You Win! You may stop training now via KeyboardInterrupt.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "#record sessions\n",
    "import gym.wrappers\n",
    "video_env = gym.wrappers.Monitor(gym.make(\"CartPole-v0\"),directory=\"videos\",force=True)\n",
    "sessions = [generate_session(video_env) for _ in range(100)]\n",
    "env.close()\n",
    "#upload to gym\n",
    "#gym.upload(\"./videos/\",api_key=\"<your_api_key>\") #you'll need me later"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#show video\n",
    "from IPython.display import HTML\n",
    "import os\n",
    "\n",
    "video_names = list(filter(lambda s:s.endswith(\".mp4\"),os.listdir(\"./videos/\")))\n",
    "\n",
    "HTML(\"\"\"\n",
    "<video width=\"640\" height=\"480\" controls>\n",
    "  <source src=\"{}\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\".format(\"./videos/\"+video_names[-1])) #this may or may not be _last_ video. Try other indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework part I\n",
    "\n",
    "### Tabular crossentropy method\n",
    "\n",
    "You may have noticed that the taxi problem quickly converges from -100 to a near-optimal score and then descends back into -50/-100. This is in part because the environment has some innate randomness. Namely, the starting points of passenger/driver change from episode to episode.\n",
    "\n",
    "### Tasks\n",
    "- __1.1__ (1 pts) Find out how the algorithm performance changes if you change different percentile and different n_samples.\n",
    "- __1.2__ (2 pts) Tune the algorithm to end up with positive average score.\n",
    "\n",
    "It's okay to modify the existing code.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```<Describe what you did here.  Preferably with plot/report to support it.>```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework part II\n",
    "\n",
    "### Deep crossentropy method\n",
    "\n",
    "By this moment you should have got enough score on [CartPole-v0](https://gym.openai.com/envs/CartPole-v0) to consider it solved (see the link). It's time to upload the result and get to something harder.\n",
    "\n",
    "* if you have any trouble with CartPole-v0 and feel stuck, feel free to ask us or your peers for help.\n",
    "\n",
    "### Tasks\n",
    "\n",
    "* __2.1__ Go to the [gym site](http://gym.openai.com/), register and obtain __api key__.\n",
    "* __2.2__ (1 pt) Upload your result to gym via gym.upload (see Results tab above, the line you need is commented)\n",
    "* __2.3__ (3 pts) Pick one of environments: MountainCar-v0 or LunarLander-v2.\n",
    "  * For MountainCar, get average reward of __at least -150__\n",
    "  * For LunarLander, get average reward of __at least +50__\n",
    "  * For any environment, upload it to gym and post url in your anytask form.\n",
    "\n",
    "See the tips section below, it's kinda important.\n",
    "__Note:__ If your agent is below the target score, you'll still get most of the points depending on the result, so don't be afraid to submit it.\n",
    "  \n",
    "  \n",
    "* __2.4__ (bonus: 1++ pt) Devise a way to speed up training at least 2x against the default version\n",
    "  * Obvious improvement: use [joblib](https://www.google.com/search?client=ubuntu&channel=fs&q=joblib&ie=utf-8&oe=utf-8)\n",
    "  * Try re-using samples from 3-5 last iterations when computing threshold and training\n",
    "  * Experiment with amount of training iterations and learning rate of the neural network (see params)\n",
    "  * __Please list what you did in anytask submission form__\n",
    "  \n",
    "  \n",
    "### Tips\n",
    "* Gym page: [mountaincar](https://gym.openai.com/envs/MountainCar-v0), [lunarlander](https://gym.openai.com/envs/LunarLander-v2)\n",
    "* Sessions for MountainCar may last for 10k+ ticks. Make sure ```t_max``` param is at least 10k.\n",
    " * Also it may be a good idea to cut rewards via \">\" and not \">=\". If 90% of your sessions get reward of -10k and 20% are better, than if you use percentile 20% as threshold, R >= threshold __fails cut off bad sessions__ whule R > threshold works alright.\n",
    "* _issue with gym_: Some versions of gym limit game time by 200 ticks. This will prevent cem training in most cases. Make sure your agent is able to play for the specified __t_max__, and if it isn't, try `env = gym.make(\"MountainCar-v0\").env` or otherwise get rid of TimeLimit wrapper.\n",
    "* If you use old _swig_ lib for LunarLander-v2, you may get an error. See this [issue](https://github.com/openai/gym/issues/100) for solution.\n",
    "* If it won't train it's a good idea to plot reward distribution and record sessions: they may give you some clue. If they don't, call course staff :)\n",
    "* 20-neuron network is probably not enough, feel free to experiment.\n",
    "* __Please upload the results to openai gym and send links to all submissions in the e-mail__\n",
    "\n",
    "### Bonus tasks\n",
    "\n",
    "* __2.4 bonus__ Try to find a network architecture and training params that solve __both__ environments above (_Points depend on implementation. If you attempted this task, please mention it in anytask submission._)\n",
    "\n",
    "* __2.5 bonus__ Solve continuous action space task with `MLPRegressor` or similar.\n",
    "  * [MountainCarContinuous-v0](https://gym.openai.com/envs/MountainCarContinuous-v0), [LunarLanderContinuous-v2](https://gym.openai.com/envs/LunarLanderContinuous-v2) \n",
    "  * 4 points for solving. Slightly less for getting some results below solution threshold. Note that discrete and continuous environments may have slightly different rules aside from action spaces.\n",
    "\n",
    "\n",
    "If you're still feeling unchallenged, consider the project (see other notebook in this folder)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}