sparse.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Разреженные признаки и Vowpal Wabbit\n",
    "\n",
    "### 21.03.2017\n",
    "\n",
    "#### Мельник Богдан"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "%pylab inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams[\"figure.figsize\"] = (20, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import scipy.sparse\n",
    "\n",
    "from IPython.display import Image, display\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer, HashingVectorizer\n",
    "from sklearn.model_selection import GridSearchCV, cross_val_predict, cross_val_score, train_test_split, KFold\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import roc_auc_score, roc_curve, f1_score, make_scorer\n",
    "from sklearn.preprocessing import MultiLabelBinarizer\n",
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "from sklearn.feature_selection import SelectKBest\n",
    "from sklearn.decomposition import TruncatedSVD\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.naive_bayes import MultinomialNB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Примеры источников разреженных признаков\n",
    "\n",
    "* Результат one-hot-encoding для большого количества категориальных признаков\n",
    "* Тексты и любые последовательности, которые можно интерпретировать как \"bag of words\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Разреженные матрицы\n",
    "\n",
    "Существует много типов разреженных матриц, каждый из которых предоставляет разные гарантии на операции.\n",
    "\n",
    "* ```scipy.sparse.bsr_matrix```\n",
    "* ```scipy.sparse.coo_matrix```\n",
    "* ```scipy.sparse.csc_matrix```\n",
    "* ```scipy.sparse.csr_matrix```\n",
    "* ```scipy.sparse.dia_matrix```\n",
    "* ```scipy.sparse.dok_matrix```\n",
    "* ```scipy.sparse.lil_matrix```\n",
    "\n",
    "Подробнее про [устройство разреженых матрицы](http://www.netlib.org/utk/people/JackDongarra/etemplates/node372.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## scipy.sparse.coo_matrix\n",
    "\n",
    "* Используется как хранилище данных\n",
    "* Поддерживает быструю конвертацию в любой формат\n",
    "* Не поддерживает индексацию\n",
    "* Поддерживает ограниченый набор арифметических операций"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "coo = scipy.sparse.random(10000, 10000, format='coo')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "'coo_matrix' object does not support indexing",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-125-355d36e159e0>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcoo\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m: 'coo_matrix' object does not support indexing"
     ]
    }
   ],
   "source": [
    "print(coo[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## scipy.sparse.csr_matrix\n",
    "\n",
    "* Хранит данные построчно\n",
    "* Быстрое получение значений отдельных строк"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## scipy.sparse.csc_matrix\n",
    "\n",
    "* Хранит данные поколоночно\n",
    "* Быстрое получение значений отдельных колонок"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 loop, best of 3: 4.44 s per loop\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "\n",
    "m = scipy.sparse.csr_matrix((10000, 10000))\n",
    "for _ in range(10000):\n",
    "    m[np.random.randint(0, 10000), np.random.randint(0, 10000)] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## scipy.sparse.lil_matrix\n",
    "\n",
    "* Отлично подходит для создания разреженных матриц поэлементно\n",
    "* Для последующих матричных операций лучше сконвертировать в ```csr_matrix``` или ```csc_matrix```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 loops, best of 3: 100 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "\n",
    "m = scipy.sparse.lil_matrix((10000, 10000))\n",
    "for _ in range(10000):\n",
    "    m[np.random.randint(0, 10000), np.random.randint(0, 10000)] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Работа с разреженными данными\n",
    "\n",
    "Библиотека ```scipy.sparse``` содержит методы, позволяющие работать с разреженными матрицами.\n",
    "\n",
    "Например, специальные ```hstack``` и ```vstack``` для конкатенации.\n",
    "\n",
    "Подробнее про операции с разрежеными матрицами на сайте [scipy](https://docs.scipy.org/doc/scipy/reference/sparse.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Что можно использовать?\n",
    "\n",
    "Подходят почти все модели\n",
    "* LogisticRegression\n",
    "* LinearSVC\n",
    "* MultinomialNB\n",
    "* RandomForestClassifier\n",
    "* KNeighborsClassifier\n",
    "\n",
    "Не подходят\n",
    "* GradientBoostingClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Пример с текстами\n",
    "\n",
    "* Датасет StackOverflow\n",
    "    * Случайная выборка из 10 тысяч вопросов\n",
    "    * Только 20 популярных тегов\n",
    "* По тексту нужно определить все теги для каждого вопроса"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 2)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>tags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>i want to use a track bar to change a form s ...</td>\n",
       "      <td>c# winforms type-conversion decimal opacity</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>i have an absolutely positioned div containin...</td>\n",
       "      <td>html css css3 internet-explorer-7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>given a datetime representing a person s birt...</td>\n",
       "      <td>c# .net datetime</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>given a specific datetime value how do i disp...</td>\n",
       "      <td>c# datetime datediff relative-time-span</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  \\\n",
       "0   i want to use a track bar to change a form s ...   \n",
       "1   i have an absolutely positioned div containin...   \n",
       "2   given a datetime representing a person s birt...   \n",
       "3   given a specific datetime value how do i disp...   \n",
       "\n",
       "                                          tags  \n",
       "0  c# winforms type-conversion decimal opacity  \n",
       "1            html css css3 internet-explorer-7  \n",
       "2                             c# .net datetime  \n",
       "3      c# datetime datediff relative-time-span  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "texts = pd.read_csv('multi_tag.10k.tsv', header=None, sep='\\t')\n",
    "texts.columns = ['text', 'tags']\n",
    "print(texts.shape)\n",
    "texts.head(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Оценка качества\n",
    "\n",
    "* Будем использовать обобщение $F1$-меры на несколько классов\n",
    "\n",
    "\n",
    "$$\n",
    "F1 = 2\\frac{p * r}{p + r},\\ \\ \\mathrm{где}\\ \\ p = \\frac{tp}{tp+fp},\\ \\  r = \\frac{tp}{tp+fn}\n",
    "$$\n",
    "\n",
    "\n",
    "$tp$, $fp$ и $fp$ считаются по всем тегам одного объекта"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5\n",
      "0.5\n"
     ]
    }
   ],
   "source": [
    "y, y_hat = np.array([[1, 1, 0, 0]]), np.array([[1, 0, 1, 0]])\n",
    "tp, fp, fn = 1., 1., 1.\n",
    "p, r = tp / (tp + fp), tp / (tp + fn)\n",
    "f1 = 2 * p * r / (p + r)\n",
    "\n",
    "print(f1)\n",
    "print(f1_score(y, y_hat, average='samples'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Bag of words\n",
    "\n",
    "Построим модель BOW с помощью ```CountVectorizer```\n",
    "* Для каждого отдельного слова будет создана колонка\n",
    "* В пересечении строки и колонки будет стоять 1, если это слово присутствует в документе\n",
    "* ```CountVectorizer``` возвращает ```scipy.sparse.csr.csr_matrix```\n",
    "* Если рассматривать вхождение каждого отдельного слова как признак, то мы получим $35247$ признаков"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 35247)\n",
      "<class 'scipy.sparse.csr.csr_matrix'>\n"
     ]
    }
   ],
   "source": [
    "vectorizer = CountVectorizer(binary=True)\n",
    "bow = vectorizer.fit_transform(texts.text)\n",
    "print(bow.shape)\n",
    "print(type(bow))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Отбор тегов\n",
    "\n",
    "* Оставим только самые популярные 20 тегов для предсказания\n",
    "* Если у вопроса после такой фильтрации не осталось тегов, то присвоим ему отдельный тег __other__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1198, 'c#'),\n",
       " (1090, '.net'),\n",
       " (696, 'java'),\n",
       " (610, 'asp.net'),\n",
       " (472, 'sql-server'),\n",
       " (462, 'c++'),\n",
       " (434, 'javascript'),\n",
       " (372, 'sql'),\n",
       " (363, 'php'),\n",
       " (349, 'windows')]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags = texts.tags.apply(lambda x: x.split())\n",
    "all_tags = reduce(lambda s, x: s + x, tags, [])\n",
    "values, count = np.unique(all_tags, return_counts=True)\n",
    "\n",
    "top_tags = sorted(zip(count, values), reverse=True)[:20]\n",
    "top_tags[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def filter_tags(tags):\n",
    "    intersection = list(good_tags.intersection(tags))\n",
    "    if intersection:\n",
    "        return intersection\n",
    "    return ['other']\n",
    "\n",
    "good_tags = set(map(lambda x: x[1], top_tags))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Преобразуем списки тегов в матрицу, которая будет содержать индикаторы наличия тега у вопроса.\n",
    "\n",
    "Для одного вопроса может быть несколько единиц."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 21)\n",
      "[[0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
      " [0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
      " [1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
      " [0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
      " [0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0]]\n"
     ]
    }
   ],
   "source": [
    "binarizer = MultiLabelBinarizer()\n",
    "y = binarizer.fit_transform(texts.tags.apply(lambda x: filter_tags(x.split())))\n",
    "print(y.shape)\n",
    "print(y[:5, :])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Выбор модели и оценка\n",
    "\n",
    "* Будет использовать ```LogisticRegression``` вместе с ```OneVsRestClassifier``` для получения предсказаний"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "params = {'estimator__C': np.logspace(-5, 5, 11)}\n",
    "\n",
    "clf = OneVsRestClassifier(LogisticRegression())\n",
    "cv = GridSearchCV(clf, params, n_jobs=-1, scoring=make_scorer(f1_score, average='samples'), cv=5)\n",
    "cv.fit(bow, y);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Выбор модели и оценка\n",
    "\n",
    "* $F1 = 0.4047323$\n",
    "* Как можно улучшить качество модели?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>params</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.404732</td>\n",
       "      <td>{u'estimator__C': 100000.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.404212</td>\n",
       "      <td>{u'estimator__C': 10000.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.404140</td>\n",
       "      <td>{u'estimator__C': 1000.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.403066</td>\n",
       "      <td>{u'estimator__C': 100.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.402630</td>\n",
       "      <td>{u'estimator__C': 10.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.390346</td>\n",
       "      <td>{u'estimator__C': 1.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.319707</td>\n",
       "      <td>{u'estimator__C': 0.1}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.092920</td>\n",
       "      <td>{u'estimator__C': 0.01}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.000300</td>\n",
       "      <td>{u'estimator__C': 0.001}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>{u'estimator__C': 1e-05}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>{u'estimator__C': 0.0001}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    mean_test_score                       params\n",
       "10         0.404732  {u'estimator__C': 100000.0}\n",
       "9          0.404212   {u'estimator__C': 10000.0}\n",
       "8          0.404140    {u'estimator__C': 1000.0}\n",
       "7          0.403066     {u'estimator__C': 100.0}\n",
       "6          0.402630      {u'estimator__C': 10.0}\n",
       "5          0.390346       {u'estimator__C': 1.0}\n",
       "4          0.319707       {u'estimator__C': 0.1}\n",
       "3          0.092920      {u'estimator__C': 0.01}\n",
       "2          0.000300     {u'estimator__C': 0.001}\n",
       "0          0.000000     {u'estimator__C': 1e-05}\n",
       "1          0.000000    {u'estimator__C': 0.0001}"
      ]
     },
     "execution_count": 205,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(cv.cv_results_)[['mean_test_score', 'params']].sort_values('mean_test_score', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Tf-Idf\n",
    "\n",
    "* Вместо бинарного значения, для каждого слова считать его вес в документе.\n",
    "* Обычно работает лучше, чем ```CountVectorizer```\n",
    "* Реализовано в ```TfidfVectorizer```\n",
    "\n",
    "Значение Tf-Idf для каждого пары документ-слово состоит из двух компонент:\n",
    "* Term frequency — логарифм встречаемости слова в документе\n",
    "$$tf(t, d) = \\log n_{t,d}$$\n",
    "\n",
    "* Inverse Document frequency — логарифм обратной доли документов в которых встретилось данное слово\n",
    "$$idf(t, D) = \\log \\frac{ \\mid D \\mid}{\\mid \\{ d_i \\in D \\mid t \\in d_i \\} \\mid}$$\n",
    "\n",
    "* Tf-Idf — кобминация tf и idf\n",
    "$$ TfIdf(t, d, D) = tf(t, d) * idf(t, D)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 35247)\n",
      "<class 'scipy.sparse.csr.csr_matrix'>\n"
     ]
    }
   ],
   "source": [
    "vectorizer = TfidfVectorizer()\n",
    "tf_idf = vectorizer.fit_transform(texts.text)\n",
    "print(tf_idf.shape)\n",
    "print(type(tf_idf))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Выбор модели и оценка\n",
    "\n",
    "Ищем лучшие параметры"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "params = {'estimator__C': np.logspace(-5, 5, 11)}\n",
    "\n",
    "clf = OneVsRestClassifier(LogisticRegression())\n",
    "cv = GridSearchCV(clf, params, n_jobs=-1, scoring=make_scorer(f1_score, average='samples'), cv=5)\n",
    "cv.fit(tf_idf, y);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Выбор модели и оценка\n",
    "\n",
    "* $F1=0.382173$\n",
    "* Получилось хуже, чем BOW.\n",
    "* Но почему?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>params</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.382173</td>\n",
       "      <td>{u'estimator__C': 100000.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.380033</td>\n",
       "      <td>{u'estimator__C': 10000.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.376427</td>\n",
       "      <td>{u'estimator__C': 1000.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.364680</td>\n",
       "      <td>{u'estimator__C': 100.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.334247</td>\n",
       "      <td>{u'estimator__C': 10.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.182323</td>\n",
       "      <td>{u'estimator__C': 1.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.000700</td>\n",
       "      <td>{u'estimator__C': 0.1}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>{u'estimator__C': 1e-05}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>{u'estimator__C': 0.0001}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>{u'estimator__C': 0.001}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>{u'estimator__C': 0.01}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    mean_test_score                       params\n",
       "10         0.382173  {u'estimator__C': 100000.0}\n",
       "9          0.380033   {u'estimator__C': 10000.0}\n",
       "8          0.376427    {u'estimator__C': 1000.0}\n",
       "7          0.364680     {u'estimator__C': 100.0}\n",
       "6          0.334247      {u'estimator__C': 10.0}\n",
       "5          0.182323       {u'estimator__C': 1.0}\n",
       "4          0.000700       {u'estimator__C': 0.1}\n",
       "0          0.000000     {u'estimator__C': 1e-05}\n",
       "1          0.000000    {u'estimator__C': 0.0001}\n",
       "2          0.000000     {u'estimator__C': 0.001}\n",
       "3          0.000000      {u'estimator__C': 0.01}"
      ]
     },
     "execution_count": 255,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(cv.cv_results_)[['mean_test_score', 'params']].sort_values('mean_test_score', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Выбор порога\n",
    "\n",
    "* При вызове ```predict``` возвращается 1, если вероятность принадлежности к классу больше $0.5$\n",
    "* Можно выбирать порог самому через кросс-валидацию"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 15.4 s, sys: 109 ms, total: 15.5 s\n",
      "Wall time: 15.6 s\n",
      "CPU times: user 10.9 s, sys: 55.4 ms, total: 11 s\n",
      "Wall time: 11 s\n"
     ]
    }
   ],
   "source": [
    "clf = OneVsRestClassifier(LogisticRegression(C=100000))\n",
    "%time y_hat_bow = cross_val_predict(clf, bow, y, method='predict_proba')\n",
    "%time y_hat_tf_idf = cross_val_predict(clf, tf_idf, y, method='predict_proba')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Функция, которая в зависимости от порога ставит тег"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def get_score(alpha, y, y_hat):\n",
    "    return f1_score(y, (y_hat > alpha).astype('int'), average='samples')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Выбор порога для BOW\n",
    "\n",
    "* Качество с порогом по умолчанию — $F1=0.404732$\n",
    "* Качество с подобранным порогом — $F1=0.454356$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.454356904762\n",
      "0.00535353535354\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABJkAAAFACAYAAAAfw61rAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuUpOddH/jvr3u6Z6bHtiTjOZHRxZKJsBHG2GYwEIjD\nSWwQh2B5E0JEDhtD2OiwsYOzxElMLiYrDgmXhGxCHIIPca6wWmxYMktEDBtgD5fY0RgbhGQUZPki\nCZvI1g13j7qmqp/9o97qru5pjVrqqem3Zj6fc+rUe61+6vJWv+/3/b1PVWstAAAAALAfCwfdAAAA\nAADmn5AJAAAAgH0TMgEAAACwb0ImAAAAAPZNyAQAAADAvgmZAAAAANg3IRMAAAAA+yZkAgAAAGDf\nhEwAAAAA7Nuhg27A+fKCF7ygXXfddQfdDAAAAICLxgc+8IFPt9aO72XZiyZkuu6663Lq1KmDbgYA\nAADARaOqPr7XZV0uBwAAAMC+CZkAAAAA2DchEwAAAAD7JmQCAAAAYN+ETAAAAADsm5AJAAAAgH0T\nMgEAAACwb0ImAAAAAPZNyAQAAADAvgmZAAAAANi3QwfdAAAADsbPfvCh/NB7783vP3Y6n3v50fyN\nr31J3vDKqw66WQDAnBIyAQBcgn72gw/lu3/mrpw+M0qSPPTY6Xz3z9yVJIImAOBZETIBwEVgY6Nl\n1Fo2WktryUZrGW20bLSktfH9Rjc/LVlYqBxaqCwsVBarsrhQWdi8T6rqoJ/SroajjZwZtQxGGznT\n3SZa277s9GjrXo8zo5bhxkaGo5Yzo40MN7r70WT+RlqSSrJQlaqMb5kMVyo7pmW8wvT45PXbGn/6\n59ZaMtxo47Z1bRyONnJmY3w/mT7aGL/HW3979zZNxjdaS+sev2X8WUhr+Ue/8N83A6aJ02dG+fsn\n787aYJSF2noNFqqysDC+T7ZP3/xbm8O1bd3JZ6sqWazxZ24yf3KrnX9r8nhT45PP56GFyf1CFhen\nx2vXz22b2iYm28FkfHGhsrS4kMWFC/d5n7Rn1H0mRxstw42t4VG3LY8mn9WNlsFwI+vD8ed9MHU/\n2BxvGW2Mt4W2+Xe2/t709CTda7aw+R0w/RoudreF7jXZev264Z3T0rJY49fx0GJleXEhhxYXsrQ4\nnra0Y/jQYmVpobvv5vX1+waAZ26mIVNV3ZTknyZZTPLjrbXvf4rl/myS9yT50tbaqaq6LsmHk9zb\nLfK+1tp3zLKtwHwajqZ2vCcHnzt2xM90B5PTB6PJ2QfR02ObO9HdyOSgZHp662ZuTR/vbE8vt5un\nOkidNGfzILM7AJ4cVA83uvtueHLQMb6Nxvdnxq/D+pnR5rynasdO4wPcjWx096NR2/UgaKO1HOoO\nEA51BymHugO9Q4vT4+ODh0MLC1k6tJClhZ0HIVsHHePn3R1Ub/t7k4On8XjrDgonB6qTcGTzoKgq\ni92B8ORAbaNbd6PtPIgbv0dLiws5fGhyW8zhpanhQwvd+GIWavy+bmyM3//NA67J5yPdQWwXZEwH\nGJMQYzDa2Hz/Js9rNNW24UbbCou6+8lrMtz8PEx/FrYCiI29vc17ttCFA5VnePC34/M8+Yx3s7aC\nhO79qx0BwiRUaGk5M+zCpOHGZqh0vp8nZ3vs9Jn87f/7roNuxrMy+dwm2RYoPZ2qjL+PFipLhxZy\naGEhy4uT4fF3y/R2ubGRrTBosh132/LG9P+CqWBvEvZytknANfnfcGhxYVsQuTOsnA4eJwHo5Ptk\nEpRP5k2+V5YXF7J8qLtNDx9ayOGp8UMLC5sh22JNhfELW99Ti107RhvZ+lxMB4TdZ2E46oL9zmbw\nl7OnTV6H6fYtLS7sOj75TE6e98Lk/9+O12oSDE/C4IWq1ML21+/sfZGtb/1JeAwXu8n3c2tt6gST\nz/6zNbOQqaoWk7wjyeuSPJjkzqo62Vq7Z8dyz03yliTv3/EQH2mtvWJW7YO+a218oDoJDyYHytMH\npOP7bO3MTHZwp3Z0Nja6ioap6eMdnqmdsIWtHYytM8tbZ5EnO8nZ3DmamtYNb7R2duixGXhshSCD\n4cZmWDBp/+bw5gHBuJ2Tx3vyzChPntnIk91jjsdHeXI4PqN/qdoMRpbGYcjydDByaCHPO7qU5cWF\nLD6Dn3iY7FxPbjvPck92ZodTZ9h3hh+TUOTMVKCydnrUhWVb1RhnhlthWdXW35hUFUyfUZ9Mq8pT\nbAtdQDa1PUx2sg8tbrV7smO+FUpl23Y2/Vl98sxoXweEk7Bt8+z9Qm07o7+4sL1tk8Bs+dDCVtun\nXv9D3QHw5ABsaRLqTVUFTCo5th2AbTvg2tppOjt0m3xHZNvB9DOx+3dFm5q39X2xdduqMtoZCiwv\nLmTpUGV5cbG73zrIWlochwCLC+OD0Ymd+4TTIdmhxa3P0+T92BaIdtOnn89WFdD2Co7N78EdVUJT\nT39b8LxXm+HttmqPqWmTbSHZ9e/uFnZPH0guVDarrr7hR34tn3riybPacOXzjuQ/vvkrtyp/Nqaq\n4zbfn8l7Nx3CbwUrk9dttPl9v2O4tanv++07+JuB7Y4gd/p/3aj7zpkOpKcD6skB9eQzv7lNTAWZ\nlWwGudMnJIajjQymhs90QcHZVX/J4sL4O3b6QH/6/+fkNd9e9TWet/37dSGLlSwuLmwGG9Pff0uL\nTxE+TE2bvM/jz/1ke6gd49lWRTXcaFMnFLa+yyffCdOVaVvVfN3zmtp3mLwfg6nXbBKwn5k6MXJm\nR0g+qdKa/O+YDuK3viuyLcDbrVpz8n0yqfLb+qxsZH043r8Y7qj+mtzWu3HObRJCTf6vbG0L2XGi\nZyvg2gzVpr4Dd6usS87+zG77u92kxV1Dx60AsKqyuEs4UDseb2v6jiBuOqDrHmcrUNz6DG4NTwLm\nre/HyfPdfH47nu/ktZh8J2ztW2Xbazh5fbf2w6fXP/vv7KxYnrRpY9v0yYmyrfnJ7t+3u+3r73wv\nd76f0yHq1vDZ62wFmmefiJreR5kcJ0x//08H/RutbXuM6X2f7Bif/D/a9hhtqwp8t//TVds/czvD\n2Z2f+83b1DaysFD5tj92Xb7pS685+w9cxGZZyfTqJPe11u5Pkqq6PcnNSe7Zsdz3JvmBJH9jhm3h\nErSx0bI+3MjpM6PxbTAOJibDp7ugYv3MxuYXUSXbdpp27lgNJo83tf5keHp8uDH95T81vOOAY9S1\ncX04DlGmD3SfHI6e0YFJ3y111SvjA+iFzR2CnWfaps8+Lh2qHDm0mJXlQ3n+sXGYcuTQYo50lSVH\nlhZyZCpgGR90jg9Klxa3nwGcDimSs//hTe/utDbZCdnlcpDpf4aTf2Q75m1VcUxdSpPpf7RPUf3U\ntl6rzQPhXQ6Al7ozvM6wzN6ZrlJuvQucFqa/F2r752N6J8TlH8yDt33dS7f1yZQkR5cW87ave2n+\nyPOOHGDL4MKanNibhGRnV6llW4XSJMxanKpwOrSwsBkUTFfZTv5nTOwM/sbTJtXE7ZyXRG5VaG/s\nGnhM2jkJb0fdzsXOcG5bgNctP71fkmwPqzMVOmw/cTl1wmJz2tZBe0090UngP71ftHOfbGcQs/Ok\nxdmh4y4nK3ecHXqqqrHJ/tf4Ndw4Kwiffp+nT1xtVspNQoRuP3ZpYWHHPmKd9Tyng7TtJ4zH792Z\n7rM3fRJo2vRnZ3pfdPKXJieVpoORycnDs0OSybTpS8Nr2z7vws7P6i7v5fbnteN9nlpx+3u9vdJz\n6/3YmrYVstVZQf50EJhsXbbbNj8b2wO00UbbdqJh52NsDW+FejtPhm1uNxvbPx/nqmQcddvIc45c\nej0UzfIZX5XkganxB5N82fQCVfWqJNe01v5TVe0Mma6vqg8meSLJ322t/erOP1BVtya5NUmuvfba\n89l2emYw3MgfPPFkPvXEk/nMZwd5/PQgj62dyWOnz+SxtTN54vSZPDaZtnYmj58+k8+uD2ferqpk\nZWkxR7rb0eVx8DG5/GfzjOXU8tVdS1IZl+Y/98ihzaBkEpwcXtqqRjmytLitsmPb2aPN4a2zIbv1\nr7K4sHU2ZjI92bmjsXWWfmPqS7R1ewnb+h6ZPK8d4+P2jp/H8tTlR5PKDJg3k7DyOYcvvR0ELn6T\nzr39uhyXuqrK8qFxtRgA+3Nge81VtZDkh5N86y6zP5nk2tbaZ6rqS5L8bFV9YWvtiemFWmvvTPLO\nJDlx4sRFVPNxaZkESJ98/Ml88vHT+dTjW8Of7IY//dn1Xat6Di1ULl9ZymVHl3L5ynKufN6RvOTK\n5+byo8t5zpFDWVlezNGl8e3I1PDR5XH4cXR5HOYkOevsziRwGY+PE/LDh6Yfb1who1oBgHn1hlde\nJVQCAM6bWYZMDyWZvvjw6m7axHOTvCzJr3QH6VcmOVlVr2+tnUqyniSttQ9U1UeSfH6SUzNsLzOw\nuj7MJx9/Mp96fFyF9KnHT3f34/DoD554Mp/+7OCs9Z57+FBeePmRXHnZ0XzBlc/LlZcdyedefiR/\n5HlHcvy5h3P5ynIuP7qUleVFIQ8AAAD0wCxDpjuT3FBV12ccLt2S5C9MZrbWHk/ygsl4Vf1Kkrd2\nvy53PMkjrbVRVb04yQ1J7p9hW3mWTg9GeeixtTzwyOk88OhaHnz0dB54pLt/dC2PrZ05a50rVpZy\n5WVH88LLjuTlV1+eF152JFc+70heePmRvPCycZD03CNLB/BsAAAAgGdrZiFTa21YVW9O8t4ki0ne\n1Vq7u6puS3KqtXbyHKu/JsltVXUmyUaS72itPTKrtvL0njwzyu9+6g9z10OP564HH8t//4PP5sFH\nT+fTn13fttzyoYVcfcXRXHPFSl5+9WW5+oqVzQqkSYB0ZGnxgJ4FAAAAMCvVduvoZg6dOHGinTrl\narrzYX04yu9+chIoPZ67Hno8//0P/nDzF9Oef2w5L73yubn2+SvjQGlyf8VKXvCcw1nQwTMAAABc\nFKrqA621E3tZ1s/lkCR54JG1/Pv3fTy/ft+nc++ntgKlK1aW8rKrLsutL3lxXn71ZXnZVZflqsuP\n6gcJAAAA2EbIdIn74CcezY//6kfz87/zySxU5ctf/Dn5y695cV5+1ThQuvoKgRIAAADw9IRMl6DR\nRssv3vOp/PivfjSnPv5onnvkUP7ya16cb/1j1+WFlx096OYBAAAAc0jIdAlZXR/mPR94MO/69Y/m\n459ZyzXPP5rv+YYb800nrsmxwz4KAAAAwLMnWbgE/METT+bf/MbH8pPv/0QeP30mr7r28rztppfm\na77wyizqpBsAAAA4D4RMF7HRRsu7fu2j+aFfuDfD0UZuetmV+favenG+5EVXHHTTAAAAgIuMkOki\n9fHPrOat7/6t3PmxR/O6G/9I/t7X35hrP2floJsFAAAAXKSETBeZjY2Wn3j/x/MP7vjdHFqs/PA3\nfXH+p1de5RfiAAAAgJkSMl1EHnrsdP7me34rv37fZ/Kazz+eH/izX+TX4gAAAIALQsh0EWit5d2n\nHsxtP3dPWmv5h3/mi3LLl16jegkAAAC4YIRMc+4Pnngyb/vp384v3/twvvzFz88PfeMX55rn63sJ\nAAAAuLCETHOqtZaTv/X7eft/vDvrw1G+5xtuzBu/4rosLKheAgAAAC48IdOcuvNjj+Ytt38or7r2\n8vyjP/fFefHx5xx0kwAAAIBLmJBpTn3ikbUkyT/586/Iiz7n2AG3BgAAALjULRx0A3h2Tg+GSZJj\nh+WEAAAAwMETMs2p1cEoSXJsWcgEAAAAHDwh05xaWx+mKjmy5C0EAAAADp6EYk6tDkZZWVpMlV+T\nAwAAAA6ekGlOrQ1GWdEfEwAAANATQqY5tTYY5tjy4kE3AwAAACCJkGlura6PsqLTbwAAAKAnhExz\nam0wzIpKJgAAAKAnhExzSp9MAAAAQJ8ImeaUPpkAAACAPhEyzSl9MgEAAAB9ImSaU/pkAgAAAPpk\npiFTVd1UVfdW1X1V9bZzLPdnq6pV1Ympad/drXdvVX3tLNs5j1YHo6wcFjIBAAAA/TCz662qajHJ\nO5K8LsmDSe6sqpOttXt2LPfcJG9J8v6paTcmuSXJFyb53CT/b1V9fmttNKv2zpPhaCOD4UaOuVwO\nAAAA6IlZVjK9Osl9rbX7W2uDJLcnuXmX5b43yQ8keXJq2s1Jbm+trbfWPprkvu7xSLJ2Zpy1uVwO\nAAAA6ItZhkxXJXlgavzBbtqmqnpVkmtaa//pma7brX9rVZ2qqlMPP/zw+Wn1HFhbn4RMKpkAAACA\nfjiwjr+raiHJDyf568/2MVpr72ytnWitnTh+/Pj5a1zPrQ6GSZJj+mQCAAAAemKWpTAPJblmavzq\nbtrEc5O8LMmvVFWSXJnkZFW9fg/rXtJOD1QyAQAAAP0yy0qmO5PcUFXXV9Vyxh15n5zMbK093lp7\nQWvtutbadUnel+T1rbVT3XK3VNXhqro+yQ1J/tsM2zpXVte7SiZ9MgEAAAA9MbNSmNbasKrenOS9\nSRaTvKu1dndV3ZbkVGvt5DnWvbuqfirJPUmGSd7kl+W2rE0qmQ6rZAIAAAD6YaYpRWvtjiR37Jj2\n9qdY9qt3jH9fku+bWePm2KRPJr8uBwAAAPTFgXX8zbO3WckkZAIAAAB6Qsg0h9Y2+2RyuRwAAADQ\nD0KmObS62SeTSiYAAACgH4RMc2htMMziQmV50dsHAAAA9IOUYg6tro+ysryYqjropgAAAAAkETLN\npdODkf6YAAAAgF4RMs2h1cFQf0wAAABArwiZ5tCaSiYAAACgZ4RMc2h1fZijyyqZAAAAgP4QMs2h\n02dGOSZkAgAAAHpEyDSHVteHWTnscjkAAACgP4RMc2jcJ5NKJgAAAKA/hExzaHV9mBUdfwMAAAA9\nImSaQ2uDUVZUMgEAAAA9ImSaM4PhRoYbLcf0yQQAAAD0iJBpzqwNhkmikgkAAADoFSHTnFkdjJII\nmQAAAIB+ETLNmbX1SSWTy+UAAACA/hAyzZm1rpLp2GGVTAAAAEB/CJnmzOpAJRMAAADQP0KmObO2\n3lUyCZkAAACAHhEyzZlJJdNRHX8DAAAAPSJkmjOn9ckEAAAA9JCQac6sdiGTPpkAAACAPhEyzZm1\n9UnH3yqZAAAAgP4QMs2Z1cEoy4sLWVr01gEAAAD9MdOkoqpuqqp7q+q+qnrbLvO/o6ruqqoPVdWv\nVdWN3fTrqup0N/1DVfUvZ9nOebI2GGZFf0wAAABAz8ysY5+qWkzyjiSvS/Jgkjur6mRr7Z6pxX6y\ntfYvu+Vfn+SHk9zUzftIa+0Vs2rfvFobjHJMf0wAAABAz8yykunVSe5rrd3fWhskuT3JzdMLtNae\nmBo9lqTNsD0XhbXBUH9MAAAAQO/MMmS6KskDU+MPdtO2qao3VdVHkvxgku+cmnV9VX2wqv6/qvrj\nu/2Bqrq1qk5V1amHH374fLa9t1bXR1k5rJIJAAAA6JcD7z26tfaO1trnJflbSf5uN/mTSa5trb0y\nyXcl+cmqet4u676ztXaitXbi+PHjF67RB2htMMzKkkomAAAAoF9mGTI9lOSaqfGru2lP5fYkb0iS\n1tp6a+0z3fAHknwkyefPqJ1zZW0wyjEdfwMAAAA9M8uQ6c4kN1TV9VW1nOSWJCenF6iqG6ZGvz7J\n73XTj3cdh6eqXpzkhiT3z7Ctc2NtMMqKjr8BAACAnplZWtFaG1bVm5O8N8likne11u6uqtuSnGqt\nnUzy5qp6bZIzSR5N8sZu9dckua2qziTZSPIdrbVHZtXWebK6PlTJBAAAAPTOTEtiWmt3JLljx7S3\nTw2/5SnW++kkPz3Lts2rtcEoR5dUMgEAAAD9cuAdf7N3rbWsDlQyAQAAAP0jZJoj68ONtBZ9MgEA\nAAC9I2SaI6vrwyRRyQQAAAD0jpBpjqwNRkmSo0tCJgAAAKBfhExzZHUwqWRyuRwAAADQL0KmOTKp\nZFpZVskEAAAA9IuQaY6srY9DJpVMAAAAQN8ImebI5HI5lUwAAABA3wiZ5sjaZsikkgkAAADoFyHT\nHJn0yXRMJRMAAADQM0KmOTLpk2lFn0wAAABAzwiZ5sikT6ajSyqZAAAAgH4RMs2RtcEoR5YWsrhQ\nB90UAAAAgG2ETHNkdX2YYzr9BgAAAHpIyDRHTg9GWTnsUjkAAACgf4RMc2R1oJIJAAAA6Cch0xxZ\nG4yysqySCQAAAOgfIdMcWV0fZkUlEwAAANBDQqY5opIJAAAA6Csh0xxZG4xy7LBKJgAAAKB/hExz\nZG0wVMkEAAAA9JKQaY6srrtcDgAAAOgnIdOc2NhoOX1mpONvAAAAoJf2HDJV1dGqesksG8NTO31m\nlCQ5dlglEwAAANA/ewqZquobknwoyX/uxl9RVSdn2TC2Wx0Mk0QlEwAAANBLe61k+vtJXp3ksSRp\nrX0oyfUzahO7WFtXyQQAAAD0115DpjOttcd3TGtPt1JV3VRV91bVfVX1tl3mf0dV3VVVH6qqX6uq\nG6fmfXe33r1V9bV7bOdFa1LJdHRJJRMAAADQP3sNme6uqr+QZLGqbqiqH0nyG+daoaoWk7wjydcl\nuTHJN0+HSJ2fbK19UWvtFUl+MMkPd+vemOSWJF+Y5KYk/6J7vEvW6YFKJgAAAKC/9hoy/dWMA5/1\nJD+Z5PEkf+1p1nl1kvtaa/e31gZJbk9y8/QCrbUnpkaPZas66uYkt7fW1ltrH01yX/d4l6zVLmTS\nJxMAAADQR0+bWHQVRLe11t6a5O88g8e+KskDU+MPJvmyXR7/TUm+K8lykj85te77dqx71TP42xed\ntfXx5XIqmQAAAIA+etpKptbaKMlXzaoBrbV3tNY+L8nfSvJ3n8m6VXVrVZ2qqlMPP/zwbBrYE5uV\nTPpkAgAAAHpor4nFB6vqZJJ3J1mdTGyt/cw51nkoyTVT41d3057K7Ul+9Jms21p7Z5J3JsmJEyee\ntiPyeXa66/h7RSUTAAAA0EN77ZPpSJLPZHw52zd0tz/9NOvcmeSGqrq+qpYz7sj75PQCVXXD1OjX\nJ/m9bvhkkluq6nBVXZ/khiT/bY9tvShNKpmO6ZMJAAAA6KE9JRattW97pg/cWhtW1ZuTvDfJYpJ3\ntdburqrbkpxqrZ1M8uaqem2SM0keTfLGbt27q+qnktyTZJjkTd1le5estfVhqpIjS3vNBQEAAAAu\nnD2FTFV1dZIfSfKV3aRfTfKW1tqD51qvtXZHkjt2THv71PBbzrHu9yX5vr2071KwOhhlZWkxVXXQ\nTQEAAAA4y17LYv51xpewfW53+3+6aVwga4NhVg67VA4AAADop72GTMdba/+6tTbsbv8myfEZtosd\n1gajHFvW6TcAAADQT3sNmT5TVd9SVYvd7Vsy7gicC2R1fZQVnX4DAAAAPbXXkOkvJfmmJJ9K8skk\n35jkGXcGzrO3Nhjm2GGVTAAAAEA/7fXX5T6e5PUzbgvnsDoY5bKjSwfdDAAAAIBd7amSqar+bVVd\nPjV+RVW9a3bNYqfTg6E+mQAAAIDe2uvlci9vrT02GWmtPZrklbNpErvRJxMAAADQZ3sNmRaq6orJ\nSFU9P3u81I7zQ59MAAAAQJ/tNSj6x0n+a1W9O0ll3PH3982sVZxldTDKUZfLAQAAAD21146//11V\nnUryJ7tJf6a1ds/smsW04Wgjg+FGjrlcDgAAAOipPaUWVfV5ST7SWrunqr46yWur6ven+2lidtbO\njJIkKyqZAAAAgJ7aa59MP51kVFV/NMmPJbkmyU/OrFVss7Y+DpmOHVbJBAAAAPTTXkOmjdbaMMmf\nSfLPW2t/I8kLZ9cspq0OhklUMgEAAAD9tdeQ6UxVfXOSv5jk57ppS7NpEjtNKplW9MkEAAAA9NRe\nQ6ZvS/IVSb6vtfbRqro+yb+fXbOYttZVMh1TyQQAAAD01F5/Xe6eJN+ZJFX1qtbabyb5gVk2jC1r\ng66SSZ9MAAAAQE/ttZJp2o+f91ZwTqsqmQAAAICeezYhU533VnBOkz6ZjgqZAAAAgJ56NiHT/37e\nW8E5bfXJ5HI5AAAAoJ+eccjUWvvZJKmql57/5rCb1c0+mVQyAQAAAP30bCqZJn7hvLWCc1obDHNo\nobK8uJ+3CwAAAGB2znn9VVX9s6ealeTy898cdrO6PsrR5cVU6Q4LAAAA6Ken6+Tn25L89STru8z7\n5vPfHHZzejDSHxMAAADQa0+XXNyZ5Hdaa7+xc0ZV/f2ZtIizrA6G+mMCAAAAeu3pQqZvTPLkbjNa\na9ef/+awmzWVTAAAAEDPPV1P0s9pra1dkJbwlFbXh1lZVskEAAAA9NfThUw/Oxmoqp9+pg9eVTdV\n1b1VdV9VvW2X+d9VVfdU1W9X1X+pqhdNzRtV1Ye628ln+rcvJmuDkZAJAAAA6LWnuwZr+ufMXvxM\nHriqFpO8I8nrkjyY5M6qOtlau2dqsQ8mOdFaW6uq/zXJDyb589280621VzyTv3mxWhsMs3J45aCb\nAQAAAPCUnq6SqT3F8F68Osl9rbX7W2uDJLcnuXnbg7f2y1OX470vydXP8G9cEsZ9MqlkAgAAAPrr\n6UKmL66qJ6rqD5O8vBt+oqr+sKqeeJp1r0rywNT4g920p/LtSX5+avxIVZ2qqvdV1Rue5m9d1MZ9\nMun4GwAAAOivcyYXrbULUj5TVd+S5ESSPzE1+UWttYeq6sVJfqmq7mqtfWTHercmuTVJrr322gvR\n1AOhTyYAAACg756ukmk/HkpyzdT41d20barqtUn+TpLXt9bWJ9Nbaw919/cn+ZUkr9y5bmvtna21\nE621E8ePHz+/re+JwXAjw42WY4dVMgEAAAD9NcuQ6c4kN1TV9VW1nOSWJNt+Ja6qXpnkxzIOmP7H\n1PQrqupwN/yCJF+ZZLrD8EvG2mCYJCqZAAAAgF6bWXlMa21YVW9O8t4ki0ne1Vq7u6puS3KqtXYy\nyQ8leU6Sd1dVknyitfb6JF+Q5MeqaiPjIOz7d/wq3SVjdTBKkhzTJxMAAADQYzNNLlprdyS5Y8e0\nt08Nv/Yp1vuNJF80y7bNi7X1rpLpsEomAAAAoL9mebkc58GkksnlcgAAAECfCZl6bqtPJpfLAQAA\nAP0lZOoFi135AAAQoElEQVS5tXV9MgEAAAD9J2TqudWBPpkAAACA/hMy9dyaPpkAAACAOSBk6rmt\nkMnlcgAAAEB/CZl6bm190vG3SiYAAACgv4RMPbc6GGX50EKWFr1VAAAAQH9JLnpubTBUxQQAAAD0\nnpCp59YGoxzTHxMAAADQc0KmnlPJBAAAAMwDIVPPra6PsnJYJRMAAADQb0KmnlsbDHNMJRMAAADQ\nc0KmnltdH7lcDgAAAOg9IVPPnT4zyoqOvwEAAICeEzL13Or6MMcOq2QCAAAA+k3I1HNrA5VMAAAA\nQP8JmXqstZbVwVCfTAAAAEDvCZl6bH24kdaikgkAAADoPSFTj62uD5NEn0wAAABA7wmZemxtMEqi\nkgkAAADoPyFTj60OukomfTIBAAAAPSdk6rHV9XEl01EhEwAAANBzQqYeO91dLnfssMvlAAAAgH4T\nMvXY5HK5FZVMAAAAQM8JmXpsbbNPJpVMAAAAQL/NNGSqqpuq6t6quq+q3rbL/O+qqnuq6rer6r9U\n1Yum5r2xqn6vu71xlu3sq0mfTCqZAAAAgL6bWchUVYtJ3pHk65LcmOSbq+rGHYt9MMmJ1trLk7wn\nyQ926z4/yfck+bIkr07yPVV1xaza2leTPplW9MkEAAAA9NwsK5leneS+1tr9rbVBktuT3Dy9QGvt\nl1tra93o+5Jc3Q1/bZJfbK090lp7NMkvJrlphm3tpUmfTEeXVDIBAAAA/TbLkOmqJA9MjT/YTXsq\n357k55/luheltcEoR5cWs7hQB90UAAAAgHPqxXVYVfUtSU4k+RPPcL1bk9yaJNdee+0MWnawVteH\nOXZYFRMAAADQf7OsZHooyTVT41d307apqtcm+TtJXt9aW38m67bW3tlaO9FaO3H8+PHz1vC+OD0Y\n5ahOvwEAAIA5MMuQ6c4kN1TV9VW1nOSWJCenF6iqVyb5sYwDpv8xNeu9Sb6mqq7oOvz+mm7aJWV1\nMMyx5V4UmwEAAACc08wSjNbasKrenHE4tJjkXa21u6vqtiSnWmsnk/xQkuckeXdVJcknWmuvb609\nUlXfm3FQlSS3tdYemVVb+2ptMMqKSiYAAABgDsy0TKa1dkeSO3ZMe/vU8GvPse67krxrdq3rv3Gf\nTCqZAAAAgP6b5eVy7NPk1+UAAAAA+k7I1GNrg5FKJgAAAGAuCJl6bG0w1CcTAAAAMBeETD22uq6S\nCQAAAJgPQqae2thoOX1Gn0wAAADAfBAy9dTpM6MkybHDQiYAAACg/4RMPbU6GCZJVpZdLgcAAAD0\nn5Cpp9bWVTIBAAAA80PI1FMqmQAAAIB5ImTqqdODcSXTyrJKJgAAAKD/hEw9tboZMqlkAgAAAPpP\nyNRTa+vjy+X0yQQAAADMAyFTT00qmY6pZAIAAADmgJCpp9a6jr+P6pMJAAAAmANCpp5aU8kEAAAA\nzBEhU0+trQ9TlRxZ8hYBAAAA/SfB6KnVwSjHlg+lqg66KQAAAABPS8jUU2uDYVb0xwQAAADMCSFT\nT60NRkImAAAAYG4ImXpqdX2UFZ1+AwAAAHNCyNRTa4Nhjh1WyQQAAADMByFTT60OVDIBAAAA80PI\n1FNr6zr+BgAAAOaHkKmn1lQyAQAAAHNEyNRT+mQCAAAA5omQqaf0yQQAAADMk5mGTFV1U1XdW1X3\nVdXbdpn/mqr6zaoaVtU37pg3qqoPdbeTs2xn3wxHGxkMN/TJBAAAAMyNmZXKVNViknckeV2SB5Pc\nWVUnW2v3TC32iSTfmuStuzzE6dbaK2bVvj5bOzNKEiETAAAAMDdmeT3Wq5Pc11q7P0mq6vYkNyfZ\nDJlaax/r5m3MsB1zZ219HDIdO+xyOQAAAGA+zPJyuauSPDA1/mA3ba+OVNWpqnpfVb3h/Dat31YH\nwyQqmQAAAID50edSmRe11h6qqhcn+aWququ19pHpBarq1iS3Jsm11157EG2cic1KJh1/AwAAAHNi\nlpVMDyW5Zmr86m7anrTWHuru70/yK0leucsy72ytnWitnTh+/Pj+WtsjayqZAAAAgDkzy5DpziQ3\nVNX1VbWc5JYke/qVuKq6oqoOd8MvSPKVmerL6WK3Nug6/tYnEwAAADAnZhYytdaGSd6c5L1JPpzk\np1prd1fVbVX1+iSpqi+tqgeT/LkkP1ZVd3erf0GSU1X1W0l+Ocn37/hVuovapE+mYyqZAAAAgDkx\n01KZ1todSe7YMe3tU8N3ZnwZ3c71fiPJF82ybX026ZNJJRMAAAAwL2Z5uRzP0uavyy2pZAIAAADm\ng5Cph7b6ZBIyAQAAAPNByNRDa4NhDi1Ulhe9PQAAAMB8kGL00Or6KCvLi6mqg24KAAAAwJ4ImXpo\nbTDMMZ1+AwAAAHNEyNRDa4NRji7rjwkAAACYH0KmHlobjHJsWSUTAAAAMD+ETD20uj7MikomAAAA\nYI4ImXpobTDSJxMAAAAwV4RMPbQ2GOqTCQAAAJgrQqYeGvfJJGQCAAAA5oeQqYfGfTK5XA4AAACY\nH0KmnmmtdX0yqWQCAAAA5oeQqWcGo40MN5pKJgAAAGCuCJl65vRglCRZ0ScTAAAAMEeETD2z2oVM\nx1QyAQAAAHNEyNQza+vDJMmKPpkAAACAOSJk6hmVTAAAAMA8EjL1zNpgXMl0VJ9MAAAAwBwRMvXM\n2rpKJgAAAGD+CJl6ZnWgTyYAAABg/giZemZNn0wAAADAHBIy9czquj6ZAAAAgPkjZOqZ010l04qQ\nCQAAAJgjQqaeWR2MsnxoIUuL3hoAAABgfkgyemZtMMwxVUwAAADAnJlpyFRVN1XVvVV1X1W9bZf5\nr6mq36yqYVV94455b6yq3+tub5xlO/vkr3z1H81/+F++7KCbAQAAAPCMzOwnzKpqMck7krwuyYNJ\n7qyqk621e6YW+0SSb03y1h3rPj/J9yQ5kaQl+UC37qOzam9fXHnZkVx52ZGDbgYAAADAMzLLSqZX\nJ7mvtXZ/a22Q5PYkN08v0Fr7WGvtt5Ns7Fj3a5P8YmvtkS5Y+sUkN82wrQAAAADswyxDpquSPDA1\n/mA3bdbrAgAAAHCBzXXH31V1a1WdqqpTDz/88EE3BwAAAOCSNcuQ6aEk10yNX91NO2/rttbe2Vo7\n0Vo7cfz48WfdUAAAAAD2Z5Yh051Jbqiq66tqOcktSU7ucd33Jvmaqrqiqq5I8jXdNAAAAAB6aGYh\nU2ttmOTNGYdDH07yU621u6vqtqp6fZJU1ZdW1YNJ/lySH6uqu7t1H0nyvRkHVXcmua2bBgAAAEAP\nVWvtoNtwXpw4caKdOnXqoJsBAAAAcNGoqg+01k7sZdm57vgbAAAAgH4QMgEAAACwb0ImAAAAAPbt\noumTqaoeTvLxg27HefKCJJ8+6EbAHLCtwN7YVmBvbCuwN7YV2JuLZVt5UWvt+F4WvGhCpotJVZ3a\na6dacCmzrcDe2FZgb2wrsDe2FdibS3FbcbkcAAAAAPsmZAIAAABg34RM/fTOg24AzAnbCuyNbQX2\nxrYCe2Nbgb255LYVfTIBAAAAsG8qmQAAAADYNyETAAAAAPsmZLoAquqmqrq3qu6rqrftMv9wVf1f\n3fz3V9V1U/O+u5t+b1V97V4fE+bR+d5Wquqaqvrlqrqnqu6uqrdcuGcDszGL/yndvMWq+mBV/dzs\nnwXM3oz2vy6vqvdU1e9W1Yer6isuzLOB2ZnRtvK/dftev1NV/2dVHbkwzwZm59luK1X1Od0xyWer\n6p/vWOdLququbp1/VlV1YZ7N7AiZZqyqFpO8I8nXJbkxyTdX1Y07Fvv2JI+21v5okn+S5Ae6dW9M\nckuSL0xyU5J/0R0E7OUxYa7MYltJMkzy11trNyb58iRvsq0wz2a0nUy8JcmHZ/sM4MKY4bbyT5P8\n59baS5N8cWwzzLkZHatcleQ7k5xorb0syWK3HMyt/WwrSZ5M8veSvHWXh/7RJH85yQ3d7abz3/oL\nS8g0e69Ocl9r7f7W2iDJ7Ulu3rHMzUn+bTf8niR/qkswb05ye2ttvbX20ST3dY+3l8eEeXPet5XW\n2idba7+ZJK21P8z4YOCqC/BcYFZm8T8lVXV1kq9P8uMX4DnAhXDet5WquizJa5L8qyRprQ1aa49d\ngOcCszST/ytJDiU5WlWHkqwk+f0ZPw+YtWe9rbTWVltrv5Zx2LSpql6Y5Hmttfe18S+y/bskb5jp\ns7gAhEyzd1WSB6bGH8zZB7mby7TWhkkeT/I551h3L48J82YW28qmrlz1lUnefx7bDBfarLaT/yPJ\n30yycf6bDAdiFtvK9UkeTvKvu0tLf7yqjs2m+XDBnPdtpbX2UJJ/lOQTST6Z5PHW2i/MpPVw4exn\nWznXYz74NI85d4RMwEWvqp6T5KeT/LXW2hMH3R7ok6r600n+R2vtAwfdFui5Q0leleRHW2uvTLKa\nRL+YsENVXZFxRcf1ST43ybGq+paDbRVwoQiZZu+hJNdMjV/dTdt1ma6k9LIknznHunt5TJg3s9hW\nUlVLGQdMP9Fa+5mZtBwunFlsJ1+Z5PVV9bGMS7//ZFX9h1k0Hi6gWWwrDyZ5sLU2qYh9T8ahE8yz\nWWwrr03y0dbaw621M0l+Jskfm0nr4cLZz7Zyrse8+mkec+4ImWbvziQ3VNX1VbWccad3J3csczLJ\nG7vhb0zyS901mSeT3NL1Un99xh2B/bc9PibMm/O+rXT9BfyrJB9urf3wBXkWMFvnfTtprX13a+3q\n1tp13eP9UmvNGWfm3Sy2lU8leaCqXtKt86eS3DPrJwIzNotjlU8k+fKqWun2xf5UdJLP/NvPtrKr\n1tonkzxRVV/ebSt/Mcl/PP9Nv7AOHXQDLnattWFVvTnJezP+ZYV3tdburqrbkpxqrZ3M+CD431fV\nfUkeSffrC91yP5XxDswwyZtaa6Mk2e0xL/Rzg/NpFttKVX1Vkv85yV1V9aHuT/3t1todF/bZwfkx\nq/8pcLGZ4bbyV5P8RHeAcX+Sb7ugTwzOsxltK++vqvck+c1u+geTvPNCPzc4n/azrSRJVzH+vCTL\nVfWGJF/TWrsnyV9J8m+SHE3y891trtU5gjUAAAAA2BOXywEAAACwb0ImAAAAAPZNyAQAAADAvgmZ\nAAAAANg3IRMAAAAA+yZkAgCYgar6WFW9YL/LAADMCyETAAAAAPsmZAIA2Keq+tmq+kBV3V1Vt+6Y\nd11V/W5V/URVfbiq3lNVK1OL/NWq+s2ququqXtqt8+qq+q9V9cGq+o2qeskFfUIAAM+CkAkAYP/+\nUmvtS5KcSPKdVfU5O+a/JMm/aK19QZInkvyVqXmfbq29KsmPJnlrN+13k/zx1tork7w9yT+YaesB\nAM4DIRMAwP59Z1X9VpL3JbkmyQ075j/QWvv1bvg/JPmqqXk/091/IMl13fBlSd5dVb+T5J8k+cJZ\nNBoA4HwSMgEA7ENVfXWS1yb5itbaFyf5YJIjOxZr5xhf7+5HSQ51w9+b5Jdbay9L8g27PB4AQO8I\nmQAA9ueyJI+21ta6PpW+fJdlrq2qr+iG/0KSX9vDYz7UDX/reWklAMCMCZkAAPbnPyc5VFUfTvL9\nGV8yt9O9Sd7ULXNFxv0vncsPJvmHVfXBbFU3AQD0WrW2s3obAIDzpaquS/Jz3aVvAAAXLZVMAAAA\nAOybSiYAAAAA9k0lEwAAAAD7JmQCAAAAYN+ETAAAAADsm5AJAAAAgH0TMgEAAACwb/8/d5myYTgf\nkZIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x110ea4bd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "alphas = np.linspace(0.0, 0.01, 100)\n",
    "scores = [get_score(alpha, y, y_hat_bow) for alpha in alphas]\n",
    "\n",
    "plot(alphas, scores);\n",
    "scatter(alphas[np.argmax(scores)], np.max(scores));\n",
    "ylabel('F1-score')\n",
    "xlabel('alpha')\n",
    "\n",
    "\n",
    "print(np.max(scores))\n",
    "print(alphas[np.argmax(scores)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Выбор порога для Tf-Idf\n",
    "\n",
    "* Качество с порогом по умолчанию — $F1=0.382173$\n",
    "* Качество с подобранным порогом — $F1=0.493972$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.493972857143\n",
      "0.00191919191919\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABJkAAAFACAYAAAAfw61rAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X+cW3d95/v3W5rR2J5xHCceyA8nsRMMIVDAMA2EFsql\nSTDLNsml7G5g2Q0tbbaFFHqB3ibLXuiaZW+he9m9hbQly6bbH9AsEBpcGppyC/TxYGnATpwm2Ik3\nzg9im4Q42LE9si2NpM/9Q0ejM/LYljNzZs6ZeT0f0UM6v+SPRjoz0Vuf71eOCAEAAAAAAAAzUZrv\nAgAAAAAAAFB8hEwAAAAAAACYMUImAAAAAAAAzBghEwAAAAAAAGaMkAkAAAAAAAAzRsgEAAAAAACA\nGSNkAgAAAAAAwIwRMgEAAAAAAGDGCJkAAAAAAAAwYwPzXcBsWbVqVaxZs2a+ywAAAAAAAFgw7rnn\nnmciYrSffRdMyLRmzRpt2bJlvssAAAAAAABYMGz/sN99GS4HAAAAAACAGSNkAgAAAAAAwIwRMgEA\nAAAAAGDGCJkAAAAAAAAwY4RMAAAAAAAAmDFCJgAAAAAAAMxYpiGT7Q22d9jeafvGaba/y/Ze2/cl\nl19JbbvO9sPJ5bos6wQAAAAAAMDMDGR1x7bLkm6WdIWk3ZI2294UEdt7dv0fEXFDz7FnSPqopDFJ\nIeme5Nj9WdULAAAAAACA5y7LTqZLJe2MiEcjoi7pNklX93nsmyR9IyL2JcHSNyRtyKhOAAAAAAAA\nzFCWIdO5knallncn63r9ou37bX/Z9nmneCwAAAAAAAByILPhcn36K0l/ERE12/9G0p9IemO/B9u+\nXtL1knT++ednUyGAOXfH1j36vbt26EfPHtE5py/Vb73pRbpmPTkzAAAAAORZlp1MeySdl1penayb\nFBE/iYhasvg5Sa/q99jk+FsiYiwixkZHR2etcADz546te3TTVx7QnmePKCTtefaIbvrKA7pj6zG/\nAgAAAAAAOZJlJ9NmSetsr1U7ILpW0jvSO9g+OyKeTBavkvRgcvsuSf/R9spk+UpJN2VYK4CMRIRq\njZYO15uq1ho6MtFUbaKlerOleqOlifR1s6Xf2bRNRyaaU+7jyERTG7+2XaPLh7RksKylg2UtrZS1\nZLCkpYNlDZZLiuTfal9LCikUitDkNkmyLUuyk9uWSql1JVuSVC5ZZVulkp/TY260Qo1mqNFqqdFs\n11W2VS6377dcal9KSR0AAAAAUHSZhUwR0bB9g9qBUVnSrRGxzfZGSVsiYpOk99m+SlJD0j5J70qO\n3Wf7Y2oHVZK0MSL2ZVUrUDSdEOPoRFO1RmvKdbXWDnMO1RqqJpdDR9vX46lLhFQZKKlSLmmwbFUG\nShosty9DqdslSxOt0ESzpYnJMCjUaLZvTzRjMjA6Um+qWm9MXh+uNXV4oqlmK2b8mPdV6/qXn/ve\nLPz0Tt1AqR02dcKhkqWBckklOxUotdrXrTjlx9sJnDr3L3VDsc49JYsKde976WBZyyoDGh7qua6U\ntWwoua4MaLBs2VbJnVArCdaS5VIStlmaDOnSgV06pItTeGih0ESzG7ZNNLs/p3qz1b7dDE20QhEx\nGQB2grd28NetzZaaLeloo6naRFNHJ1o6MtHU0clLa/L2RDNUKkkDpZLKJWug1A32epcjpFaEWsl1\ns9UOJ5utmLK+U0+5pORnaZWS2+WkzvZ9lzQ0WNKSgfIx10sGSxoaLGvJQPt6oOTkXLMGyqXu7VL7\nerBc0kC585roPheRDlF7nqNOXVYqRJ3yPE//cxiYvC5N+dn1/txK9uQ5MVOtVmgi9dqod14TzZZa\nMTWYLZXUPQdLU8Pa5xoId0TyPDdaLTVboYlGp66WJhrt1+vElEu3xsmfSXnqeTxQ7v68yiWr/Ypu\nv447Orc7r/fOus5zZ0nqWe7s24pQM3m9NluhVqtdf/s1nNxute+zU8tAuZRct2saLJVULrevS6XU\n75nk9dXq+V3Q+fXT+/ogLAcAAB2OU3nHkGNjY2OxZcuW+S4DOK5ao6kDRyZ08MiEDiSXQ0fbgU87\nAGpOhkLjkwFRU+O1hg7XG8eESaea2yyrlDU8NKCRoXYYUbZVa0x9w1RvtFJvpqaGJZWeN8KVsjU4\nUJp8k1wZKLX/jcqAlibXy4bSy+3gY1mlrKGB8mSwVencXxJs/cv/erd+fKh2TP2jI0P6zDvWp0KF\ndsBwpN7URLM15Y2Z1NOxpG5Y0QkUet+wt1JvpFoRarXab+A6141WcrvVDSIarVDJ0mC5G2S037yV\nNFjy5Ju3zpuwZkjNVkvNVs918qaws27yMajzWLqPqbM+pMlQ8XC9oWq9qcO15LremFx/uD61K2y+\n2NJgqTT5ZrcyUNJAsjxYLqUCruSNrWLyDXJ6XdnWksFyO6hJOtmWJLeXdG4PlDU40A6PGs1I/Ywj\nWU7CwAg1m9EOYDphgDshTfs5K5e6YZzUDrkiua9O+DQZRCWh1ESzdcz52nt9dKI1X0/FrOqETQOl\ndEjSfZ3qmNey1eqEjM8hkD0RW5Nh00An7OnpHOwEMhPNVvd1kFxjZrohZfc5mAw2JwNkH/M7ufO7\nu1c6UO+w2r9jK6lAtvP3YzKwHWj//i0l/1Bpyt+C1L+ZWt+v6eo8nnTw1vm3S+l/252fWbvedAA4\neTt5TFPC3tTresrPuvPz7udxJD+XgWM6a49dl/4wopR6ziaD9gwCxs7v/H7NdcjZCcc7v7/S/+8h\nTX2ttbd3/5YAQFHZvicixvrZd74n/gYK7eDRCT114Kh+9OyR9vWBo3rqwBHtq9Yng6TOpZ83lZ0g\nphMEDVcGdPaKJVo2NKAlA6XJN9NDA8deDyVvsoeTjpblSwY0PJRcKgOTHTKnotPJMTCH/3N00z95\nsW76ygNThswtHSzrw295sV594ZlzUsNC0kreQLeSDxTSnTmdjqX0cm+3xOQbQE/d1q/0myR0RcRk\n106j2QldWpMdNJ1unokkjJG63VxTurw0dZhn+7674deU4C4dirWUBG/tf6sVqcAlCeKa0e4uat8+\nfkDaaLXDOmlq510nJEh/lhUR7TfVqS6tdFgwkATYnW6qZqv7Gp4MfZPXc6fGZrO7Lb1fJxhupPbv\ndmSVpnQblUulKV1dk0F6J8wY6NY1OJB0f5ZLk911na6hTidUMxVedW53fj69XYrq+VlF6ufXWVa6\nszC5XXJP8NAJCkpTO6ik9nPV+eCgE7BNtELNaTowjwlEesL7zvM45fXSeS0lr4VG6nmaUndPoN/t\nkoq+fq+0IqacG51O2mqt0f2wpNl9TXdee90PFmLqz/gUPmg9lRiy87hareSZTX2oMRmiR/e1vRB0\ngp4p1+rplk2e41bqtdD5m5QO75+rdFdduROOpc6F3m1OrU9/uBChKR/ATSSvqZmG4+kgtlyaGiaW\nk98vUm8ncbdjtbNt8meeCgc7j6ETZk3puO15DnpDw3T42zHlw4LJdd1t5cl/a+r0Ap0PZ9Kds+lj\n0/eZPufLPY+lnHreurenC/SmhsWdv43dc61z3k39O9jZbmnyeej8Teg+R6Upz1nvfulO6fRy+md5\nzP9LpWrtDY8nPyDpWQcUESETMI3D9YaeOVTX3vGj2nuorr3jNe09VNNTB47oyQNH9eSBo3rqwFGN\n1xpTjrPbHTerRoa0YumgLlw1ohVLB7Vi2aBWLB3UaUsHddqSgfa6ZHkkCYKWDZZz98ekXLLKp/DJ\n7WzofIsc3y43O0olq5Kz1xXa/+M5NFDWEH+FgUUtHdh1wox28JceOjo1sDzmkmzvNzTrhMzpELbR\nE9Z21vW+SZ/uA4pOmNYNj7rHdJY7b+ylbvDTDT+ODXp6w4Tj//w6na+pIc/RM+S51V3XCdnT+6WH\nSXc+kBmc7LLuDF8+NhzvDm9PalEcEwZ1Hv90QWwnlO4E+xO9HygoNZQ23YknTYaY7cfRfR46w2Uj\nusNppe7j7gSczVYr1Y07NbxKP55j1k3TwdtM/fyakz/rbmA45b577q9bW/L6S93nKWTAC1pviNUJ\notJd9J3lpZWyllW6c5e2bw9oaaWkZZWByXlN7WOf5+5yV6cLrxN4pcPLqUFg6oOPyX2729MffKSD\n6ClhZyrw7bzmu3UcP6ycLpDt3J7uw4b0B3CTv6/U/b0gaUqt3U73bp2doLp3SoH0hz6LvXOR4XJY\n8CJCh+tN7T9c1/7qRPv6cF37q3XtPzyhfdW6nklCpM51dZohRp0A6ezTl+rs05bo7NOX6OwVS3T2\niqXt69OX6nnLhzRYzvJLGwEAAICFrdPd1oxu9226EzRCx+kEjWk7t9x7ndzPZNdpMxX8pbpRG0lX\nWyfsbaQ62nqHXk80W6n6p+lE09QOznTHbScU7V3X2xk73XDvzjypRyaaOlxvTyWRvn243phRlyBO\nXTqE+sAVL9T1r79ovkuaMYbLYdFpNFt6/CeHteOpQ9rx1EE9+NQh7dp3OAmUJlRvTD9UzZZWLB2c\n7D562erTtWpkSKPLh7RqpKLR5e3boyNDOmO4ogECJAAAACBTTuYIy/rN6mA5438gBzpD9I/WW1Om\no0gPM0zf6HQOpTvi0h2A6Q627vD77tD0KduTrsjJLrvUl0qkOwu7XXLdjiIpPbR8+o64KV1/qY6/\nTkdk8t8xX4rSO6y3c3zncbfS3ZCpOls923qnHOjtMm21Qi89Z8UsPpvFQMiEwtl7qKaHnjqoHU8d\n0kNPHdJDTx3Uwz8eVy0JkkqW1q4a1tpVI3r56tN1+vCgVi6r6IxlFZ2+bFBnDFd0+rKKzhiuaMXS\nQeaKAQAAALAgdYfol7VCg/NdDhYBQibkUqsV+tGBI9r59Lge2VttXz89rp17x7WvWp/cb3T5kC4+\na7n+1Wsu0MVnn6aLz1quFzxvREsWw8cSAAAAAADkCCET5t3Th47qviee1f/68SHtTIKkR56uTmnn\nPH3ZoF4wOqIrL3m+1j1/uV581nK96KzlOnNkaB4rBwAAAAAAHYRMmFONZksPPXVI9z6xX/f8cL/u\nfWK/du07Mrn9nBVLdNHzRnTtpWfoBc8b0QtGR/SC543ojOHKop+lHwAAAACAPCNkQqb2V+vauqsd\nKN3zw/36x10HJjuUnn/akF51wUpdd9karT9/pS4+a7mG+T5vAAAAAAAKiXf0mHUHjkzo6w88qb/c\nukffe2yfpPZXOL7knNP0L376PL3ygpV61QUrdc6KJXQnAQAAAACwQBAyYVbUGk1966GndcfWH+mb\nDz2terOlC0eH9ZuXr9NlF56pl60+XUsrTMYNAAAAAMBCRciE56zVCn3/8X366n179Nf3P6mDRxta\nNTKkd77mAv3v68/VS889jU4lAAAAAAAWCUImnLInfnJYf7H5CX116x796MBRLauUteElZ+ma9efq\ntRedqYFyab5LBAAAAAAAc4yQCX17/JmqPvOtnfrLrXskSa9ft0q//eaLdcUlz9eyCi8lAAAAAAAW\nM5IBnNTjz1T16W/u1B337dFAybrusjX6Nz93oZ5/2pL5Lg0AAAAAAOQEIROO67Fnqvr0Nx/WHVv3\nqDJQ0rte2w6XnreccAkAAAAAAExFyIRjPLp3XJ9JOpcqAyX98s+s1fWESwAAAAAA4AQImTDpsWeq\n+v2/e1hfTcKld//sWl3/+os0unxovksDAAAAAAA5R8gEHZ1o6uZv7dQf/f0jGiiV9Cuvu1C/+roL\nCZcAAAAAAEDfCJkWuW/veFof+eo2PbHvsN76ynN105tfTLgEAAAAAABOGSHTIvXjg0e18Wvb9df3\nP6kLR4f1hV99tV570ar5LgsAAAAAABQUIdMi02yF/uwfHtd/+tv/pYlmSx+84oW6/ucu1NBAeb5L\nAwAAAAAABUbItIjcv/tZffgvf6AH9hzQ69at0n+45qW64Mzh+S4LAAAAAAAsAKUs79z2Bts7bO+0\nfeMJ9vtF22F7LFleY/uI7fuSyx9lWedCd/DohD761R/o6pv/p3588Kg+8471+tNfvpSACQAAAAAA\nzJrMOplslyXdLOkKSbslbba9KSK29+y3XNL7JX2v5y4eiYhXZFXfYvHkgSN66x98V08dPKrrLluj\nD1z5Qp22ZHC+ywIAAAAAAAtMlsPlLpW0MyIelSTbt0m6WtL2nv0+JukTkn4rw1oWpYlmSzd8YasO\nHpnQV379tVp//sr5LgkAAAAAACxQWQ6XO1fSrtTy7mTdJNuvlHReRPz1NMevtb3V9t/bft10/4Dt\n621vsb1l7969s1b4QvGJrz+ke364X59428sImAAAAAAAQKYynZPpRGyXJH1K0gen2fykpPMjYr2k\nD0j6gu3TeneKiFsiYiwixkZHR7MtuGD+5gdP6nPfeUzXXXaB/unLzpnvcgAAAAAAwAKXZci0R9J5\nqeXVybqO5ZJeKunbth+X9BpJm2yPRUQtIn4iSRFxj6RHJL0ww1oXlMefqeq3vnS/Xn7e6fq3b3nx\nfJcDAAAAAAAWgSxDps2S1tlea7si6VpJmzobI+JARKyKiDURsUbS3ZKuiogttkeTicNl+0JJ6yQ9\nmmGtC8bRiabe8/l7VSpZN79jvYYGyvNdEgAAAAAAWAQym/g7Ihq2b5B0l6SypFsjYpvtjZK2RMSm\nExz+ekkbbU9Iakn6tYjYl1WtC8m//6tt2v7kQd36rjGtXrlsvssBAAAAAACLRJbfLqeIuFPSnT3r\nPnKcfd+Qun27pNuzrG0huv2e3fqL7+/Se95wkd548fPnuxwAAAAAALCIzNvE35hdO546pA/f8YBe\nvfYMfeAKpq8CAAAAAABzi5BpARivNfTrn79HI0OD+vTb12ugzNMKAAAAAADmFmlEwUWEbrz9fj3+\nTFWffvt6Pe+0JfNdEgAAAAAAWIQImQruz+7+ob52/5P64JUv0mUXnTnf5QAAAAAAgEWKkKnA/nHX\ns/rY17brf3vRqH795y6a73IAAAAAAMAiRshUUAeOTOg9n79Xz1u+RJ/6569QqeT5LgkAAAAAACxi\nA/NdAJ6b7zz8jPY8e0R//u5Xa+VwZb7LAQAAAAAAixydTAV16OiEJOnC0eF5rgQAAAAAAICQqbDG\naw1J0nCFZjQAAAAAADD/CJkKqlprSpKGh8rzXAkAAAAAAAAhU2FV6w0NDZQ0UOYpBAAAAAAA84+E\noqDGaw2NDDFUDgAAAAAA5AMhU0FVaw0NEzIBAAAAAICcIGQqqGqtScgEAAAAAAByg5CpoKq1hoYr\nTPoNAAAAAADygZCpoKp1hssBAAAAAID8IGQqKCb+BgAAAAAAeULIVFDtib8ZLgcAAAAAAPKBkKmg\nmPgbAAAAAADkCSFTAUWEqnWGywEAAAAAgPwgZCqgIxNNRYhOJgAAAAAAkBuETAU0XmtIkoYrzMkE\nAAAAAADyIdOQyfYG2zts77R94wn2+0XbYXsste6m5Lgdtt+UZZ1FU601JdHJBAAAAAAA8iOzlMJ2\nWdLNkq6QtFvSZtubImJ7z37LJb1f0vdS6y6RdK2kl0g6R9L/Z/uFEdHMqt4iqXY6mQiZAAAAAABA\nTmTZyXSppJ0R8WhE1CXdJunqafb7mKRPSDqaWne1pNsiohYRj0namdwf1B0ux8TfAAAAAAAgL7IM\nmc6VtCu1vDtZN8n2KyWdFxF/farHJsdfb3uL7S179+6dnaoLgE4mAAAAAACQN/M28bftkqRPSfrg\nc72PiLglIsYiYmx0dHT2isu5ar09anBkiIm/AQAAAABAPmTZCrNH0nmp5dXJuo7lkl4q6du2Jeks\nSZtsX9XHsYtap5NpWYVOJgAAAAAAkA9ZdjJtlrTO9lrbFbUn8t7U2RgRByJiVUSsiYg1ku6WdFVE\nbEn2u9b2kO21ktZJ+n6GtRYKw+UAAAAAAEDeZJZSRETD9g2S7pJUlnRrRGyzvVHSlojYdIJjt9n+\noqTtkhqS3ss3y3V1Jv4erjBcDgAAAAAA5EOmrTARcaekO3vWfeQ4+76hZ/njkj6eWXEFVq01tGSw\npIHyvE2pBQAAAAAAMAUpRQGN15oaYagcAAAAAADIEUKmAqrWGszHBAAAAAAAcoWQqYAO1xsa5pvl\nAAAAAABAjhAyFdB4raHhISb9BgAAAAAA+UHIVEDVWpPhcgAAAAAAIFcImQqIOZkAAAAAAEDeEDIV\n0HitoRHmZAIAAAAAADlCyFRAdDIBAAAAAIC8IWQqmFYrdHiiqREm/gYAAAAAADlCyFQwRyaaihCd\nTAAAAAAAIFcImQqmWmtIkpYRMgEAAAAAgBwhZCqY8SRkYrgcAAAAAADIE0KmgqnWmpKkYb5dDgAA\nAAAA5AghU8F0O5kImQAAAAAAQH4QMhVMZ04mJv4GAAAAAAB5QshUMNU6IRMAAAAAAMgfQqaCmZyT\niYm/AQAAAABAjhAyFQzD5QAAAAAAQB4RMhVMZ+Jvvl0OAAAAAADkCSFTwVRrDS0dLKtc8nyXAgAA\nAAAAMImQqWCq9QZD5QAAAAAAQO4QMhVMtdbUCJN+AwAAAACAnCFkKphqjU4mAAAAAACQP5mGTLY3\n2N5he6ftG6fZ/mu2H7B9n+3v2L4kWb/G9pFk/X22/yjLOotkvNZg0m8AAAAAAJA7maUVtsuSbpZ0\nhaTdkjbb3hQR21O7fSEi/ijZ/ypJn5K0Idn2SES8Iqv6iqpab2h0ZGi+ywAAAAAAAJgiy06mSyXt\njIhHI6Iu6TZJV6d3iIiDqcVhSZFhPQtCtdZkuBwAAAAAAMidLEOmcyXtSi3vTtZNYfu9th+R9ElJ\n70ttWmt7q+2/t/266f4B29fb3mJ7y969e2ez9twarzU0QsgEAAAAAAByZt4n/o6ImyPiIkm/Lenf\nJauflHR+RKyX9AFJX7B92jTH3hIRYxExNjo6OndFzyMm/gYAAAAAAHmUZci0R9J5qeXVybrjuU3S\nNZIUEbWI+Ely+x5Jj0h6YUZ1FkarFTpcZ7gcAAAAAADInyxDps2S1tlea7si6VpJm9I72F6XWnyL\npIeT9aPJxOGyfaGkdZIezbDWQjg80ZQkDVfK81wJAAAAAADAVJm1xEREw/YNku6SVJZ0a0Rss71R\n0paI2CTpBtuXS5qQtF/Sdcnhr5e00faEpJakX4uIfVnVWhTVWkOS6GQCAAAAAAC5k2laERF3Srqz\nZ91HUrfff5zjbpd0e5a1FdF4EjIx8TcAAAAAAMibeZ/4G/2jkwkAAAAAAOQVIVOBjE+GTMzJBAAA\nAAAA8oWQqUCqtfbE3wyXAwAAAAAAeUPIVCCH6wyXAwAAAAAA+UTIVCCTw+UqhEwAAAAAACBf+g6Z\nbC+1/aIsi8GJVZmTCQAAAAAA5FRfIZPtX5B0n6S/SZZfYXtTloXhWOPJnEx0MgEAAAAAgLzpt5Pp\ndyRdKulZSYqI+yStzagmHEe11tCySlmlkue7FAAAAAAAgCn6DZkmIuJAz7qY7WJwYtVag0m/AQAA\nAABALvWbWGyz/Q5JZdvrJL1P0nezKwvTqdabGiFkAgAAAAAAOdRvJ9NvSHqJpJqkL0g6IOk3syoK\n0+sMlwMAAAAAAMibk7bF2C5L2hgRH5L04exLwvGMM1wOAAAAAADk1Ek7mSKiKeln56AWnES11mC4\nHAAAAAAAyKV+E4uttjdJ+pKkamdlRHwlk6owLSb+BgAAAAAAedVvYrFE0k8kvTG1LiQRMs2h8VpT\nI0PMyQQAAAAAAPKnr5ApIn4p60JwctVaQ8MVOpkAAAAAAED+9PXtcrZX2/5L208nl9ttr866OHQ1\nW6EjE02GywEAAAAAgFzqK2SS9MeSNkk6J7n8VbIOc+RwvSFJGma4HAAAAAAAyKF+Q6bRiPjjiGgk\nl/8uaTTDutCjWmtKEp1MAAAAAAAgl/oNmX5i+522y8nlnWpPBI45Ml5rdzKNEDIBAAAAAIAc6jdk\n+mVJ/1zSU5KelPQ2SUwGPoeqScjExN8AAAAAACCP+v12uR9KuirjWnACkyETnUwAAAAAACCH+v12\nuT+xfXpqeaXtW/s4boPtHbZ32r5xmu2/ZvsB2/fZ/o7tS1LbbkqO22H7Tf0+oIWqWm/PycRwOQAA\nAAAAkEf9Dpd7WUQ821mIiP2S1p/oANtlSTdLerOkSyS9PR0iJb4QET8VEa+Q9ElJn0qOvUTStZJe\nImmDpD9I7m/R6nYyLeofAwAAAAAAyKl+Q6aS7ZWdBdtn6ORD7S6VtDMiHo2IuqTbJF2d3iEiDqYW\nhyVFcvtqSbdFRC0iHpO0M7m/RWuc4XIAAAAAACDH+k0s/h9J/2D7S5Ks9sTfHz/JMedK2pVa3i3p\n1b072X6vpA9Iqkh6Y+rYu3uOPXeaY6+XdL0knX/++f08jsJiTiYAAAAAAJBnfXUyRcSfSnqrpB+r\n/Q1zb42IP5uNAiLi5oi4SNJvS/p3p3jsLRExFhFjo6Ojs1FObnVCpmWDDJcDAAAAAAD50+/E3xdJ\neiQiPiPpB5IuT08Efhx7JJ2XWl6drDue2yRd8xyPXfDGa00NV8oqlTzfpQAAAAAAAByj3zmZbpfU\ntP0CSZ9VOwD6wkmO2Sxpne21titqT+S9Kb2D7XWpxbdIeji5vUnStbaHbK+VtE7S9/usdUGq1hoM\nlQMAAAAAALnVb2rRioiG7bdK+kxEfNr21hMdkOx/g6S7JJUl3RoR22xvlLQlIjZJusH25ZImJO2X\ndF1y7DbbX5S0XVJD0nsjovmcHuECUa03NELIBAAAAAAAcqrf1GLC9tsl/WtJv5CsGzzZQRFxp6Q7\ne9Z9JHX7/Sc49uM6+eTii0a11tCyIeZjAgAAAAAA+dTvcLlfknSZpI9HxGPJELZZmfgb/anWmhqu\n0MkEAAAAAADyqa/UIiK2S3qfJNl+ZUTcK+kTWRaGqcZrDZ29Ysl8lwEAAAAAADCtfjuZ0j4361Xg\npKp1Jv4GAAAAAAD59VxCJs96FTgpvl0OAAAAAADk2XMJmf79rFeBkxqvNTTCxN8AAAAAACCnTjlk\niog7JMn2xbNfDqbTbIWOTrToZAIAAAAAALn1XDqZOv521qrACVXrDUni2+UAAAAAAEBunTC1sP37\nx9sk6fRprunMAAATjklEQVTZLwfTqdaSkIlOJgAAAAAAkFMnSy1+SdIHJdWm2fb22S8H0+mGTMzJ\nBAAAAAAA8ulkIdNmST+IiO/2brD9O5lUhGOM15qSpBE6mQAAAAAAQE6dLLV4m6Sj022IiLWzXw6m\nw3A5AAAAAACQdyeb+HskIg7PSSU4rk7IRCcTAAAAAADIq5OFTHd0bti+PeNacBydb5dbVmFOJgAA\nAAAAkE8nC5mcun1hloXg+JiTCQAAAAAA5N3JQqY4zm3MIeZkAgAAAAAAeXey1OLltg+q3dG0NLmt\nZDki4rRMq4OkdshkM1wOAAAAAADk1wlDpogg1ciB8VpDw5UB2T75zgAAAAAAAPPgZMPlkAPVWkPD\nQ+R9AAAAAAAgvwiZCqBabzIfEwAAAAAAyDVCpgKoJsPlAAAAAAAA8oqQqQAYLgcAAAAAAPKOkKkA\nxmtNjTBcDgAAAAAA5BghUwG0O5kImQAAAAAAQH5lGjLZ3mB7h+2dtm+cZvsHbG+3fb/tv7N9QWpb\n0/Z9yWVTlnXmHSETAAAAAADIu8ySC9tlSTdLukLSbkmbbW+KiO2p3bZKGouIw7Z/XdInJf2LZNuR\niHhFVvUVSbXeYLgcAAAAAADItSw7mS6VtDMiHo2IuqTbJF2d3iEivhURh5PFuyWtzrCeQmo0Wzo6\n0eLb5QAAAAAAQK5lGTKdK2lXanl3su543i3p66nlJba32L7b9jXTHWD7+mSfLXv37p15xTlUrTcl\niW+XAwAAAAAAuZaL9hjb75Q0JunnUqsviIg9ti+U9E3bD0TEI+njIuIWSbdI0tjYWMxZwXOoWmtI\nEnMyAQAAAACAXMuyk2mPpPNSy6uTdVPYvlzShyVdFRG1zvqI2JNcPyrp25LWZ1hrbhEyAQAAAACA\nIsgyZNosaZ3ttbYrkq6VNOVb4myvl/RZtQOmp1PrV9oeSm6vkvQzktIThi8a40nINMJwOQAAAAAA\nkGOZtcdERMP2DZLuklSWdGtEbLO9UdKWiNgk6fckjUj6km1JeiIirpL0Ykmftd1SOwj73Z5vpVs0\nqrVkTiYm/gYAAAAAADmWaXIREXdKurNn3UdSty8/znHflfRTWdZWFNU6w+UAAAAAAED+ZTlcDrOA\nOZkAAAAAAEAREDLlXDdkYk4mAAAAAACQX4RMOTeezMk0QicTAAAAAADIMUKmnKvWGipZWjpIJxMA\nAAAAAMgvQqacG681NFwZUPLtewAAAAAAALlEyJRzh+sNJv0GAAAAAAC5R8iUc9Vak0m/AQAAAABA\n7hEy5dx4jU4mAAAAAACQf4RMOVdN5mQCAAAAAADIM0KmnKOTCQAAAAAAFAEhU85V6w2NMCcTAAAA\nAADIOUKmnGtP/E0nEwAAAAAAyDdCppyr1hoaIWQCAAAAAAA5R8iUY41mS7VGS8uY+BsAAAAAAOQc\nIVOOVWtNSdIwczIBAAAAAICcI2TKsfF6Q5IYLgcAAAAAAHKPkCnHqrV2yMTE3wAAAAAAIO8ImXJs\nvEYnEwAAAAAAKAZCphyjkwkAAAAAABQFIVOOMfE3AAAAAAAoCkKmHJvsZKrQyQQAAAAAAPKNkCnH\nqnWGywEAAAAAgGLINGSyvcH2Dts7bd84zfYP2N5u+37bf2f7gtS262w/nFyuy7LOvGLibwAAAAAA\nUBSZhUy2y5JulvRmSZdIervtS3p22yppLCJeJunLkj6ZHHuGpI9KerWkSyV91PbKrGrNq2qtoZKl\nJYM0nAEAAAAAgHzLMr24VNLOiHg0IuqSbpN0dXqHiPhWRBxOFu+WtDq5/SZJ34iIfRGxX9I3JG3I\nsNZcqtaaGh4akO35LgUAAAAAAOCEsgyZzpW0K7W8O1l3PO+W9PVTOdb29ba32N6yd+/eGZabP9Va\ng6FyAAAAAACgEHIxDsv2OyWNSfq9UzkuIm6JiLGIGBsdHc2muHlUrTe0rFKe7zIAAAAAAABOKsuQ\naY+k81LLq5N1U9i+XNKHJV0VEbVTOXahG6816WQCAAAAAACFkGXItFnSOttrbVckXStpU3oH2+sl\nfVbtgOnp1Ka7JF1pe2Uy4feVybpFpVpraJiQCQAAAAAAFEBmCUZENGzfoHY4VJZ0a0Rss71R0paI\n2KT28LgRSV9KJrd+IiKuioh9tj+mdlAlSRsjYl9WteZVtdbQGcPL5rsMAAAAAACAk8q0TSYi7pR0\nZ8+6j6RuX36CY2+VdGt21eXfOBN/AwAAAACAgsjFxN+YXnu4HBN/AwAAAACA/CNkyrFqvcmcTAAA\nAAAAoBAImXJqotlSvdHScIWQCQAAAAAA5B8hU05Vaw1JopMJAAAAAAAUAiFTTo0nIdMIczIBAAAA\nAIACIGTKqWqtKYlOJgAAAAAAUAyETDk1znA5AAAAAABQIIRMOXW43hkuR8gEAAAAAADyj5AppyYn\n/ubb5QAAAAAAQAEQMuXU+OScTEz8DQAAAAAA8o+QKaeqzMkEAAAAAAAKhJAppzoTfzMnEwAAAAAA\nKAJCppyq1hoql6yhAZ4iAAAAAACQfyQYOVWtNTRcKcv2fJcCAAAAAABwUoRMOVWtNxkqBwAAAAAA\nCoOQKaeqtYaWETIBAAAAAICCIGTKqfFag2+WAwAAAAAAhUHIlFPVWkMjQ+X5LgMAAAAAAKAvhEw5\nVa01NVyhkwkAAAAAABQDIVNOjdcaTPwNAAAAAAAKg5App6p15mQCAAAAAADFQciUU4drTUImAAAA\nAABQGIRMOVRvtFRvtjRcYeJvAAAAAABQDJmGTLY32N5he6ftG6fZ/nrb99pu2H5bz7am7fuSy6Ys\n68ybaq0hSXQyAQAAAACAwsgsxbBdlnSzpCsk7Za02famiNie2u0JSe+S9KFp7uJIRLwiq/rybDwJ\nmZj4GwAAAAAAFEWWKcalknZGxKOSZPs2SVdLmgyZIuLxZFsrwzoKp1qnkwkAAAAAABRLlsPlzpW0\nK7W8O1nXryW2t9i+2/Y10+1g+/pkny179+6dSa250h0ux5xMAAAAAACgGPI88fcFETEm6R2S/ovt\ni3p3iIhbImIsIsZGR0fnvsKMVGtNSQyXAwAAAAAAxZFlyLRH0nmp5dXJur5ExJ7k+lFJ35a0fjaL\ny7NOJ9OyCiETAAAAAAAohixDps2S1tlea7si6VpJfX1LnO2VtoeS26sk/YxSczktdEz8DQAAAAAA\niiazkCkiGpJukHSXpAclfTEittneaPsqSbL907Z3S/pnkj5re1ty+IslbbH9j5K+Jel3e76VbkFj\nTiYAAAAAAFA0mbbKRMSdku7sWfeR1O3Nag+j6z3uu5J+Ksva8qxab8/JxLfLAQAAAACAosjzxN+L\n1nitoYGSNTTA0wMAAAAAAIqBFCOHqrWGhocGZHu+SwEAAAAAAOgLIVMOVWtNJv0GAAAAAACFQsiU\nQ9VaQ8sqTPoNAAAAAACKg5Aph6r1BpN+AwAAAACAQiFkyqHxWoPhcgAAAAAAoFAImXKoPfE3w+UA\nAAAAAEBxEDLlULXWZLgcAAAAAAAoFEKmHKrWGS4HAAAAAACKhZAph9rD5QiZAAAAAABAcRAy5Uyt\n0dREMzRcYU4mAAAAAABQHIRMOVOtNSWJTiYAAAAAAFAohEw5U601JBEyAQAAAACAYiFkypnxJGRi\n4m8AAAAAAFAkhEw5QycTAAAAAAAoIkKmnKnW23MyjQwx8TcAAAAAACgOQqac6XQyLavQyQQAAAAA\nAIqDkClnmJMJAAAAAAAUESFTzjAnEwAAAAAAKCJCppzphkzMyQQAAAAAAIqDkClnxmtNDZatoQFC\nJgAAAAAAUByETDlzuN5gqBwAAAAAACicTEMm2xts77C90/aN02x/ve17bTdsv61n23W2H04u12VZ\nZ5685w0v0Od/5dXzXQYAAAAAAMApyaxlxnZZ0s2SrpC0W9Jm25siYntqtyckvUvSh3qOPUPSRyWN\nSQpJ9yTH7s+q3rw4a8USnbViyXyXAQAAAAAAcEqy7GS6VNLOiHg0IuqSbpN0dXqHiHg8Iu6X1Oo5\n9k2SvhER+5Jg6RuSNmRYKwAAAAAAAGYgy5DpXEm7Usu7k3Wzdqzt621vsb1l7969z7lQAAAAAAAA\nzEyhJ/6OiFsiYiwixkZHR+e7HAAAAAAAgEUry5Bpj6TzUsurk3VZHwsAAAAAAIA5lmXItFnSOttr\nbVckXStpU5/H3iXpStsrba+UdGWyDgAAAAAAADmUWcgUEQ1JN6gdDj0o6YsRsc32RttXSZLtn7a9\nW9I/k/RZ29uSY/dJ+pjaQdVmSRuTdQAAAAAAAMghR8R81zArxsbGYsuWLfNdBgAAAAAAwIJh+56I\nGOtn30JP/A0AAAAAAIB8IGQCAAAAAADAjBEyAQAAAAAAYMYWzJxMtvdK+uF81zFLVkl6Zr6LAAqA\ncwXoD+cK0B/OFaA/nCtAfxbKuXJBRIz2s+OCCZkWEttb+p1UC1jMOFeA/nCuAP3hXAH6w7kC9Gcx\nnisMlwMAAAAAAMCMETIBAAAAAABgxgiZ8umW+S4AKAjOFaA/nCtAfzhXgP5wrgD9WXTnCnMyAQAA\nAAAAYMboZAIAAAAAAMCMETIBAAAAAABgxgiZ5oDtDbZ32N5p+8Zptg/Z/h/J9u/ZXpPadlOyfoft\nN/V7n0ARzfa5Yvs829+yvd32Ntvvn7tHA2Qji78pybay7a22v5b9owCyl9H/f51u+8u2H7L9oO3L\n5ubRANnJ6Fz5P5L/9/qB7b+wvWRuHg2Qned6rtg+M3lPMm77Mz3HvMr2A8kxv2/bc/NoskPIlDHb\nZUk3S3qzpEskvd32JT27vVvS/oh4gaT/LOkTybGXSLpW0kskbZD0B8mbgH7uEyiULM4VSQ1JH4yI\nSyS9RtJ7OVdQZBmdJx3vl/Rgto8AmBsZniv/r6S/iYiLJb1cnDMouIzeq5wr6X2SxiLipZLKyX5A\nYc3kXJF0VNL/JelD09z1H0r6VUnrksuG2a9+bhEyZe9SSTsj4tGIqEu6TdLVPftcLelPkttflvTz\nSYJ5taTbIqIWEY9J2pncXz/3CRTNrJ8rEfFkRNwrSRFxSO03A+fOwWMBspLF3xTZXi3pLZI+NweP\nAZgLs36u2F4h6fWS/pskRUQ9Ip6dg8cCZCmTvyuSBiQttT0gaZmkH2X8OICsPedzJSKqEfEdtcOm\nSbbPlnRaRNwd7W9k+1NJ12T6KOYAIVP2zpW0K7W8W8e+yZ3cJyIakg5IOvMEx/Zzn0DRZHGuTEra\nVddL+t4s1gzMtazOk/8i6f+U1Jr9koF5kcW5slbSXkl/nAwt/Zzt4WzKB+bMrJ8rEbFH0n+S9ISk\nJyUdiIi/zaR6YO7M5Fw50X3uPsl9Fg4hE4AFz/aIpNsl/WZEHJzveoA8sf1PJT0dEffMdy1Azg1I\neqWkP4yI9ZKqkpgXE+hhe6XaHR1rJZ0jadj2O+e3KgBzhZApe3sknZdaXp2sm3afpKV0haSfnODY\nfu4TKJoszhXZHlQ7YPp8RHwlk8qBuZPFefIzkq6y/bjard9vtP3nWRQPzKEszpXdknZHRKcj9stq\nh05AkWVxrlwu6bGI2BsRE5K+Ium1mVQPzJ2ZnCsnus/VJ7nPwiFkyt5mSetsr7VdUXvSu009+2yS\ndF1y+22SvpmMydwk6dpklvq1ak8E9v0+7xMomlk/V5L5Av6bpAcj4lNz8iiAbM36eRIRN0XE6ohY\nk9zfNyOCT5xRdFmcK09J2mX7RckxPy9pe9YPBMhYFu9VnpD0GtvLkv8X+3kxST6KbybnyrQi4klJ\nB22/JjlX/rWkr85+6XNrYL4LWOgiomH7Bkl3qf3NCrdGxDbbGyVtiYhNar8J/jPbOyXtU/LtC8l+\nX1T7f2Aakt4bEU1Jmu4+5/qxAbMpi3PF9s9K+leSHrB9X/JP/duIuHNuHx0wO7L6mwIsNBmeK78h\n6fPJG4xHJf3SnD4wYJZldK58z/aXJd2brN8q6Za5fmzAbJrJuSJJScf4aZIqtq+RdGVEbJf0Hkn/\nXdJSSV9PLoXmEwRrAAAAAAAAQF8YLgcAAAAAAIAZI2QCAAAAAADAjBEyAQAAAAAAYMYImQAAAAAA\nADBjhEwAAAAAAACYMUImAACADNh+3Paqme4DAABQFIRMAAAAAAAAmDFCJgAAgBmyfYfte2xvs319\nz7Y1th+y/XnbD9r+su1lqV1+w/a9th+wfXFyzKW2/8H2Vtvftf2iOX1AAAAAzwEhEwAAwMz9ckS8\nStKYpPfZPrNn+4sk/UFEvFjSQUnvSW17JiJeKekPJX0oWfeQpNdFxHpJH5H0HzOtHgAAYBYQMgEA\nAMzc+2z/o6S7JZ0naV3P9l0R8T+T238u6WdT276SXN8jaU1ye4WkL9n+gaT/LOklWRQNAAAwmwiZ\nAAAAZsD2GyRdLumyiHi5pK2SlvTsFidYriXXTUkDye2PSfpWRLxU0i9Mc38AAAC5Q8gEAAAwMysk\n7Y+Iw8mcSq+ZZp/zbV+W3H6HpO/0cZ97ktvvmpUqAQAAMkbIBAAAMDN/I2nA9oOSflftIXO9dkh6\nb7LPSrXnXzqRT0r6v21vVbe7CQAAINcc0du9DQAAgNlie42kryVD3wAAABYsOpkAAAAAAAAwY3Qy\nAQAAAAAAYMboZAIAAAAAAMCMETIBAAAAAABgxgiZAAAAAAAAMGOETAAAAAAAAJgxQiYAAAAAAADM\n2P8Ptr5/H6xCt6gAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1034ca990>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "alphas = np.linspace(0.0, 0.01, 100)\n",
    "scores = [get_score(alpha, y, y_hat_tf_idf) for alpha in alphas]\n",
    "\n",
    "plot(alphas, scores);\n",
    "scatter(alphas[np.argmax(scores)], np.max(scores));\n",
    "ylabel('F1-score')\n",
    "xlabel('alpha')\n",
    "\n",
    "print(np.max(scores))\n",
    "print(alphas[np.argmax(scores)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## BM25\n",
    "\n",
    "Метод вычисления весов ```Okapi BM25``` развивает идею Tf-Idf, учитывая длину документов в $tf(t, d)$.\n",
    "\n",
    "$$tf(t, d) = \\frac{(K_1 + 1) * n_{t,d}}{k_1 * (1 - b + b * \\frac{L_d}{L_{ave}}) + n_{t,d}}$$\n",
    "\n",
    "где\n",
    "* $L_{ave}$ — средняя длина документа\n",
    "* $1.2 \\leqslant k \\leqslant 2$\n",
    "* $b = 0.75$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## N-граммы\n",
    "\n",
    "* При увеличении N количество признаков стремительно растёт\n",
    "* Даже для 10 тысяч объектов валидация и подбор гиперпараметров существенно замедляются"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 1132373)\n"
     ]
    }
   ],
   "source": [
    "vectorizer = CountVectorizer(binary=True, ngram_range=(1, 3))\n",
    "bow = vectorizer.fit_transform(texts.text)\n",
    "print(bow.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Чем больше признаков, тем больше мы будем тратить времени на оценку и подбор гиперпараметров."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 58s, sys: 4.28 s, total: 2min 3s\n",
      "Wall time: 2min 4s\n"
     ]
    }
   ],
   "source": [
    "%time y_hat_bow = cross_val_predict(clf, bow, y, method='predict_proba')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Оценка N-грамм\n",
    "\n",
    "* С $N \\in \\{1, \\dots, 3\\}$ и $1132373$ признаков получилось $F1 = 0.5275$\n",
    "* ```cross_val_predict``` занял ```2min 4s```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.527566904762\n",
      "0.00131313131313\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABJIAAAFACAYAAADnI1acAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xuc7GddJ/jPU9WXc7pPEjDBTCSRhCEC0dEBzgRwvRwR\nEZwdYGecWWAYxdGNuyPijJcdGHd5ucxrLuquF1Z0zeB9UAaB0ehGUdGz6ihKICgk4RIDSgIoAUI4\nXed0dVU9+0ddurrPSU6RdHX9+vT7/aJfVb9f/frXT3Wfh3P6k+/zfUqtNQAAAABwPq1FDwAAAACA\ng0GQBAAAAMBMBEkAAAAAzESQBAAAAMBMBEkAAAAAzESQBAAAAMBMBEkAAAAAzESQBAAAAMBMBEkA\nAAAAzGRp0QP4bF122WX16quvXvQw9sTGxkbW19cXPQxoPHMFZmOuwGzMFTg/8wRmcyHNlXe84x33\n1lofdb7rDlyQdPXVV+eWW25Z9DD2xMmTJ3PixIlFDwMaz1yB2ZgrMBtzBc7PPIHZXEhzpZTyl7Nc\nZ2kbAAAAADMRJAEAAAAwE0ESAAAAADMRJAEAAAAwE0ESAAAAADMRJAEAAAAwE0ESAAAAADMRJAEA\nAAAwE0ESAAAAADMRJAEAAAAwk6VFDwBIfuXWe/KDb3lfPnLf6XzeI47me7728Xn+kx696GEBAADA\nDoIkWLBfufWevOLN787prX6S5J77TucVb353kgiTAAAAaBRBEszZYFBzeqs//OgOHzvd8fNeXvVr\nt09CpLHTW/183023pT+oWWqXtFslS62SdquVpVZJq1WSJLXW4eP4E+v4YXImq0vtHFlujR7bObrS\nzpGlVo4sD4/bo3sBAADA+QiSWJhef5BPdrr5xKnRx8bm5PFTna1s9Qbp15r+YPgxmDxP+oNB+nUY\npKy0W1lfXcr6ajvrK0tZW13KsdX28NzK0uixnZWl1iReqePApdazzvUHNZ1uL51uPxvdXjqbo8du\nPxub24+nt/rZ3Bpksz9ItzdIt9fPZm/0fHJukN6g7n7rM7nv9Fa+65f/7GF9j2ex3C45sjQMmNZW\n2llbWcrayvB4ffR8bXV4/uhyO0utkq1BTa8/fG9b/UF6/ZreYJCt/vD81qCm1joJvpZaJUvt0fP2\n9vHyKBwroyxrHGlNjkdPPnhXN7fVOzMY1PQGw6/VG9T0+sM/E73BeAzD41rr8P7tkqVWK+1WGT4f\nj6HVmgR0dSp8q1M/qklINzrXmoR5O4O9dis7Ar52KWm3hmNvl5JWK2mVklYZfl4pGV1z9vdk8v0a\njXupVdJul8n1w/tkdB8BIAAAsP8ESeypM1v93HtqM/ee6ubjn9kcPv/MZj5+avx8FBhtdHNfZ+uc\n92i3Sh5xdDkrS620yvCX6uEv5Nu/ULcnv7Qnn+wNsrHZy8ZU0DMPR5Zbo6BqGLAcXWlndamVS1aW\ns9JuZXWplZWl1vD58vBxZXTu6LgSaHkY1oyPjy63862/8I78zWc2z/p6l1+8mjd869Mn4ch0aDKo\nNePY5ewQpkyOa5LNrX7O9AY5s9Wf+hhMHk9Pnd/o9nN6FJrdf6aXv77/TDrd/uijlzNbg8n4xoHH\n8iiUGQdDS+3hcauUDAY1W1Mhzzh8mg6eZvaB902ejoOccbCz3G7teEwyCbT6U2FXfzSe+tCyvUYZ\nB1K758XS9ONUkLY0CqKW2iVrK+1cdGQ5Fx1ZysWjx+HH8Pmx1eHzlXYr3f7grNBwa/S97Y2/r6Nv\naBmNq6Rk9L+UUibnW6VM5sA4rJwOLteW21lq2wMCAACaTJDEZ63XH+Suezdyx0fvz+0fvT93fPQz\nufuTnXz81GY+c6Z3zs+55OhyLju2ksuOreYJf+viXHpsJZ+zvpJLj63msvXt55eur+SSo8uTpVsP\nxWBQ09kahkrDj2FFUbc32P4lN7sCmMnz4S/hayvtSSXT+JfdeS0B+zdf98QdPZKS5OhyO694zhPz\nmEvX5/I1H6rBYBgaLO1hRUx/MK782blMb7pS6Pd///fz5V/+FZOQ5OF+7XEg1x/UlJRM365M/VmY\nPj+uiOsN6qQy6uzjQfqD7WtrTfp1WE03GNQManZV1w1DmfF9piureoOa/ijA6Y++7+e8R93++uOf\nz87gse54v1v9YcXdhz/ZyWfO9PKZM1s5tdnLQyyc23Mr7VaOjioIV0ah5PKocmtlafi43G6NPsqD\nzMud59ut4bwaL+kcB7lHlluT8+NzayvDMG19tT16HIZdqsAAAECQxHl8urM1CotGHx+7P+//61Pp\n9oaVKcvtksd97kV54hUX5ysuWs1lx1byqItWc9mx1cnjpcdWsrrU3rcxt1olx1aHvwgeBOOG2gdh\n17ZWq6SVvf1lejsIeOD7LreGlSx7+TXbrf37M9l0tdZsdPv5zJmtSbh0/5le+v26I8hZak+HOzuX\nCA7vMwwCa61Ty0e3lw32a82ZUa+wcaXb6a3epGfY5Fy3l25/XAE1rFzb6g+Gx4Oabm+QTreXrX4d\nVeftfj9nv8f+oOZMb/h1xtV43f7g7AsfQCkZLZUdhszHVpeG4dNye6risDU5N3lcaaddSmqGIeD4\nezPY9Tj+Ho0rt4Zfc7iUsWQ498rkXMnqUmv7ay61szoKxT56apCP3Hd6FJi1Jj87IRgAAHvlYPym\nzb768Cc7efM778mvvuue3HXvxuT8pesreeIVF+cbn/6YPPGKi/PEKy7O337UsawsWYrycD3/SY9u\nZHDE4VDKdvh6xSWLHs3+6Q9GwdbWdsB0equfU+NKxs3e6Hlv9Hx0rjs8Pt3t575ONx+dXiba7edM\nr//ZLdvca3/4u2edml4Oeq7+YStLrV2N+VtZXR4u3z0y9bjcbk2WKiY7K/d2nJ/0BdvZI2y6j1hr\naklme2qJ5nhp7FKrlVYrO5ZnjivRlqeq1cbPV0ahmR5iAADzJUgiSbKx2cvN7/5o3vTOu/O2uz6Z\nJHn6Yy/NPz5+VZ54xUW57oqL86iLVv3jHLhgtFtl1Kh/7/8q3OoPJsHUYJBhZVEpk4qj1miZbWmN\nKo5G/986ruga1OHBYLQhwPhcTc1gkGz2+qP+ZoMdvc/e+efvyWMf9wXD494gW6OG/71dvcq2BjX9\n/nYPs25vkM3e8B6dbi+f3Ng+3uwNNxZYeED2WWiV7QArZefx+Gcwecyot9eufl6THl8lkwBrqdXK\n8lIrK5Mwa/v5cruV6ZWW09+p3VVyNZla8rpdrTYY/cwHdbiUuGZ7+e14fOM/S63Ruuwyen8rS+1J\nj77VUTA47t23urQdDK4utXYsD11pD9/TjuP29ueNg0T9ywCAMUHSITYY1Lztrk/kje+4O7/xno/l\n9FY/V1+6lu/6mi/I//DkR+fKR64teogAB9L4F/WLjizv69c9cu97c+L6z5/r15je0bBOnavZDkzG\nS/bGPbvqjp5emfT22t3nqz+Y6j1Wh72+BnW7l9h4mWNvMMhWbxiEjcOy7ngHyf5gO6gZBTOpO4/P\nWl44fj9TSw2H4c7w3LjZ/FZ/kO54d8r+IBvd/ujrD3fp3B2zTf+nl93/IWYcbo0rqFrTIePU8fi9\nTMaX6bFvj3mrP8jmKBAcPg4my9D3wtJoifF0uDReUjne4XN9dWm04+ew19j66vbj0eWlrC61JlVn\n0xVx05sljDcGaLW2A9cHCmJbreFcW2m3HlZvRQDgsyNIOoQ+eO9G3vSOu/Nfb70n99x3OhetLuX5\nT/q8/KMnX5mnPOaRqo4AeECTnSF3/FXh740mGowCtknAtDUYBWJ1FIoNg7gdx6OPza3h553Z6k8e\nxxVq049ntoa7et57anOyw+fGZn/HBhL7YXrp48pU1dW4SX97HFCVYc+xz9x/Oj/+3j+ehFPDQO/s\nyq/pyrTpirWldmtS4TUJ1pbak4qw8bntr789jqVdO2ru3H10tPvp1FLU6V1Sdwdm2+Hn2YFuksm4\nx1V4/o0HwF4QJB0it33k03n1Wz+Qt9z212mV5MuufVT+9XOekGddd/meNjIGABav1So50mqP/o7f\n3+q4/qDm9FY/nc1eNrrD/mLjpZW90c6S07tJ9qeWYPZ3VZGdq1n9ePnfdPi1NVqmudUfTB63+sNr\nxlVx46q0zfZweeugDr9utz8cc02SqUBmsCuoSTKplpsO6PZr6ec4BzrXpgKfjemgrDUVaJ0r9Jqu\nFFtqD/uZjfudTX/upBfaZOnlOMjaDp+nj1OSdhkHa6P+bePwbHyuNd6hs3V2z7VRKNguO6vYzlUt\nOT4x3tggSVaXWzm6PKymW5vapXfyfLltSSfAAxAkHQJ/fvd9efVb78zv3PHXuejIUl72jMflRU99\nTP7WJUcWPTQA4ALUbvgOqidPnsyJE0/b03v2R7tKjqu4xssMx0s2x0s0e1NLOHcHa71RqLa1q6/Z\n9DXJqAZwusdXdlZNlVJ2VSttL+PMriWSu5eX9kY91IbHgx3nx9cN6vZS1G5vsL1cdRSyjdVdAc7u\ngGewK0jc6k+939H9F2m8U+lwzNvnR9/JHefarTLZtXMcRk3v7rm2MtzJ8+jydP+y9nYfs6lz4+Pp\nCtDJHrdTFXKT18p0oJapQG97M4NSps6PNzwY945rZfv51O6hW4OazV7/rPc6rZThpggtFW9wqDTz\nb3f2xK1/9am8+q0fyO+97+O55OhyvvNrviDf+KVX55Kj+/tfJQEALnTtVsnRUYDA3pgOuAZ1u6fa\noGZybhxq1dG56WBtbHpZ3/j0cFnmcBlmp9vP6W5vtDxzuJNnp9tPZ6uXwVSYNR2UlF1P+v1hFd7p\n0QYI43uMd/cc7xB6utsfVdIdjM0L8lu/OfOl07twLrVK2u3tHTlbDxAy7T49rpLbUXk2Xe02Dr1a\n5eylpUvDDQeOTJaZbm8WMNnxs7W98+e44m1cDTce//YOo1PnpkK5pXaZ7BY6vXmBII3DRJB0Abrl\nQ5/Mj771A/mDD9ybR64t53u+9vH5hqc/Zt+bvgIAwEPVapWsXKCN1Cc9zLbObpI/Ph43/99empft\nc9lexjesLqvpD7K9wUHdDuCmNznY3h3yHMd1+/NLKbnrrrvy2Mc+djLmHeFcyqTSbLvSbjAaw/bj\nOAg8V0VTzdknJ+MZ1PSnn0+Ndboa7lMb3R393DZ7g2yOnvf2uaJtOqwa9kdr7QqjsjOUmgrejoyq\n2cbLKndUtK20JtVuK0ut4TLTXYHXeJnp9r0zXBI6Ct+WppaHLrVak8q19qi8bTp8nfSGG72vyZLU\nqSWr4+uSs0Pa6cq2cRCoR9uFR5B0AfmTuz6RH33rB/JHf/GJXLq+kpc/5wl58dMe09iycgAAOIwW\n2cNsVifL3Tlx4nGLHsZD1httNtAb9UobLx/dGi2f3Brv9jlaVjkOqHbsLDrYGcr1B8P7bg3qZNfO\n6f5s073ZtnYs+8xZu5WO79vr13S6vXxiozupZhtXt+3l7ptNsL1D6WjnzdawOmxc2TXujzZ+Pg7m\n6o4wtU493z6fTC3vzc4Aa/q4VYYbG0xvkDBd0Ta9vHR5tGHCuEfcOKhb2hUIvufefp7e62d16fBU\npM41YSilPDvJjyZpJ3ltrfU/7nr9JUl+MMk9o1M/Vmt97TzHdCGqtebf33xH/tMffDCXHVvN937d\nE/NPn/b5WVsRIAEAAIfPsHH7wW6Y3usPcqY3mCyL3BoMdvQjG0yFXOPNCqYr0KaXho77nO0OyMa9\n05Kpqrfdx6Px1F2lZdN90JLdVXLZUfU2DoGmq9/GYd44gHugsG/crD8519LV7YqoyQ6W440aBtsb\nNkwvh+0N6lnVgJtb/ZwZ9bV7KP7Js7byuRcLkh62Uko7yWuSfE2Su5O8vZRyU6319l2X/pda60vn\nNY7D4Id/+/35T3/wwbz4aZ+f7/2666zNBwAAOOCW2q0ca7esMNlH40q28VLT6Qq13ZsNjAO5W97x\nzjxibWXRQ99X8/wTeX2SO2utdyVJKeX1SZ6XZHeQxMPwEyf/Iq/+3TvzPx6/Kq967heldYGuIwcA\nAIB5Gleyra/O/jn33zXsX3WYlN3laXt241K+Psmza63fMjr+Z0meOl19NFra9h+SfDzJ+5P8q1rr\nh89xrxuS3JAkl19++VNe//rXz2XM++3UqVM5duzYQ/783/7Lrbzujm6edkU7N3zx6gPuhgAH3cOd\nK3BYmCswG3MFzs88gdlcSHPlq77qq95Raz1+vusWXSP3a0l+qda6WUr51iQ/l+QZuy+qtd6Y5MYk\nOX78eD1x4sS+DnJeTp48mYf6Xt7w9g/ndXf8eZ513eV5zT99cpYP+PpfeDAPZ67AYWKuwGzMFTg/\n8wRmcxjnyjzTh3uSXDV1fGW2m2onSWqtn6i1bo4OX5vkKXMczwXjV991T/71m/88X/kFj8r//aIn\nCZEAAACAfTHPBOLtSa4tpVxTSllJ8oIkN01fUEq5YurwuUnumON4Lghvue1j+c43/Fmuv/pz8v+8\n+CmHaotBAAAAYLHmtrSt1torpbw0yVuStJP8dK31tlLKq5LcUmu9KcnLSinPTdJL8skkL5nXeC4E\n/9/7P55v/8Vb88VXXpKfesnfszsbAAAAsK/m2iOp1npzkpt3nXvl1PNXJHnFPMdwoXjbXZ/IDT9/\nSx73ucfys990vS0gAQAAgH2nuc4B8M6/+lS++Wffns//nLX8wjdfn0uOLi96SAAAAMAhJEhquNs/\ncn9e8tN/mssuWs3rvuWpufTY6qKHBAAAABxSgqSG+5HfeX+W26287luems+9+MiihwMAAAAcYoKk\nhrvv9FauvfxYrnzk2qKHAgAAABxygqSG63R7WV/RWBsAAABYPEFSw3U2+1mzQxsAAADQAIKkhtvo\n9rK23F70MAAAAAAESU03rEgSJAEAAACLJ0hqsFprNvRIAgAAABpCkNRgm71BBjUqkgAAAIBGECQ1\nWKfbTxIVSQAAAEAjCJIabGOzlyRZW1GRBAAAACyeIKnBJhVJqyqSAAAAgMUTJDXYRldFEgAAANAc\ngqQG62yqSAIAAACaQ5DUYCqSAAAAgCYRJDVYZxQk2bUNAAAAaAJBUoNtjJa2ra2qSAIAAAAWT5DU\nYCqSAAAAgCYRJDXYuCLp6LKKJAAAAGDxBEkN1un2cnS5nVarLHooAAAAAIKkJtvo9rOuPxIAAADQ\nEIKkBjvd7WdNfyQAAACgIQRJDbax2cvaiookAAAAoBkESQ3W6fazvqoiCQAAAGgGQVKDbXRVJAEA\nAADNIUhqsM5mP+t6JAEAAAANIUhqsI1uL2t2bQMAAAAaQpDUYJ2uiiQAAACgOQRJDbaxqSIJAAAA\naA5BUkP1+oNs9gYqkgAAAIDGECQ1VGernyR2bQMAAAAaQ5DUUJ3NcZCkIgkAAABoBkFSQ210e0mS\ndT2SAAAAgIYQJDWUiiQAAACgaQRJDdUZVyTpkQQAAAA0hCCpoTrdUUXSqookAAAAoBkESQ21oSIJ\nAAAAaBhBUkNNeiSpSAIAAAAaQpDUUCqSAAAAgKYRJDXUpEeSXdsAAACAhhAkNdTGZi/L7ZKVJT8i\nAAAAoBmkFA3V6fZVIwEAAACNIkhqqI3Nnv5IAAAAQKMIkhqq0+3bsQ0AAABoFEFSQ210e1lTkQQA\nAAA0iCCpoTqbfUESAAAA0CiCpIba6Payrtk2AAAA0CCCpIY6rUcSAAAA0DBzDZJKKc8upbyvlHJn\nKeXlD3LdPyql1FLK8XmO5yAZViRZ2gYAAAA0x9yCpFJKO8lrkjwnyXVJXlhKue4c112U5DuS/Mm8\nxnIQDXskqUgCAAAAmmOeFUnXJ7mz1npXrbWb5PVJnneO6/5tku9PcmaOYzlQaq3DiqRVFUkAAABA\nc8yz5OXRST48dXx3kqdOX1BKeXKSq2qt/28p5Xse6EallBuS3JAkl19+eU6ePLn3o12AU6dOnfO9\ndPs1g5p87O6/zMmTH93/gUHDPNBcAXYyV2A25gqcn3kCszmMc2Vha6dKKa0kP5TkJee7ttZ6Y5Ib\nk+T48eP1xIkTcx3bfjl58mTO9V4+cWoz+e3fyd954hfkxNOv3vdxQdM80FwBdjJXYDbmCpyfeQKz\nOYxzZZ5L2+5JctXU8ZWjc2MXJfmiJCdLKR9K8rQkN2m4nXS6/STRIwkAAABolHkGSW9Pcm0p5ZpS\nykqSFyS5afxirfXTtdbLaq1X11qvTvK2JM+ttd4yxzEdCBvdXpLYtQ0AAABolLkFSbXWXpKXJnlL\nkjuSvKHWelsp5VWllOfO6+teCDY2RxVJqyqSAAAAgOaYa1JRa705yc27zr3yAa49Mc+xHCQdFUkA\nAABAA81zaRsP0bgi6aggCQAAAGgQQVIDbVckWdoGAAAANIcgqYEmu7atqkgCAAAAmkOQ1EAqkgAA\nAIAmEiQ10KRH0rKKJAAAAKA5BEkN1On2srbSTqtVFj0UAAAAgAlBUgNtdPtZs6wNAAAAaBhBUgN1\nNntZ12gbAAAAaBhBUgOpSAIAAACaSJDUQJ1uL+srKpIAAACAZhEkNdDGZj9rqyqSAAAAgGYRJDWQ\niiQAAACgiQRJDbSxqUcSAAAA0DyCpAbqdHtZU5EEAAAANIwgqYE2uv2srQqSAAAAgGYRJDVMrz9I\ntzfIuqVtAAAAQMMIkhqms9VPEkvbAAAAgMYRJDVMZ3MYJK2vqkgCAAAAmkWQ1DAb3V4SFUkAAABA\n8wiSGmZSkaRHEgAAANAwgqSGmVQk2bUNAAAAaBhBUsN0RkGSiiQAAACgaWYOkkopR0spj5/nYEg2\nJs22VSQBAAAAzTJTkFRK+QdJ3pXkN0fHf7eUctM8B3ZYdSbNtlUkAQAAAM0ya0XS9yW5Psl9SVJr\nfVeSa+Y0pkNtQ7NtAAAAoKFmDZK2aq2f3nWu7vVg2K5IOrpiaRsAAADQLLOWvdxWSnlRknYp5dok\nL0vyR/Mb1uG10e1nuV2ysqQPOgAAANAss6YV357kC5NsJvnFJJ9O8i/nNajDrLPZ0x8JAAAAaKTz\nJhallHaSV9VavzvJ985/SIdbp9vPumVtAAAAQAOdtyKp1tpP8mX7MBYyDJLWVlUkAQAAAM0za2Jx\naynlpiS/nGRjfLLW+ua5jOoQ2+j2VCQBAAAAjTRrkHQkySeSPGPqXE0iSNpjnc2+HkkAAABAI82U\nWNRav2neA2Foo9vLFZccWfQwAAAAAM4y065tpZQrSyn/tZTyN6OPN5VSrpz34A6jTldFEgAAANBM\nMwVJSX4myU1JPm/08Wujc+yxjc1e1lf1SAIAAACaZ9Yg6VG11p+ptfZGHz+b5FFzHNehpSIJAAAA\naKpZg6RPlFJeXEppjz5enGHzbfZQrdWubQAAAEBjzRok/fMk/yTJx5J8NMnXJ9GAe4+d2Rqk1mRt\nVUUSAAAA0Dyz7tr2l0meO+exHHob3V6SqEgCAAAAGmnWXdt+rpTyiKnjR5ZSfnp+wzqcOpv9JMlR\nPZIAAACABpp1adsX11rvGx/UWj+V5EnzGdLh1dlSkQQAAAA016xBUquU8sjxQSnlczLjsjhmtzGq\nSNIjCQAAAGiiWROL/yvJH5dSfjlJybDZ9r+b26gOqY4eSQAAAECDzdps++dLKbckecbo1D+std4+\nv2EdTpOKJD2SAAAAgAaaKbEopfztJH9Ra729lHIiyTNLKR+Z7pvEwzepSFpVkQQAAAA0z6w9kt6U\npF9KeVySn0xyVZJfnNuoDqmNrookAAAAoLlmDZIGtdZekn+Y5Mdqrd+T5Ir5Detw6myqSAIAAACa\na9YgaauU8sIk35Dk10fnluczpMNro9tPKcmRJUESAAAA0DyzBknflOTpSf5drfWDpZRrkvzC/IZ1\nOHU2e1lbbqfVKoseCgAAAMBZZgqSaq2311pfVmv9pVLKk2utH6y1fv/5Pq+U8uxSyvtKKXeWUl5+\njtf/51LKu0sp7yql/GEp5bqH8iYuFBvdftZW9UcCAAAAmmnWiqRpr53lolJKO8lrkjwnyXVJXniO\noOgXa61/p9b6d5P8QJIfegjjuWB0ur2sr1jWBgAAADTTQwmSZl13dX2SO2utd9Vau0len+R50xfU\nWu+fOlxPUh/CeC4YG5t9O7YBAAAAjfVQUov/Y8brHp3kw1PHdyd56u6LSinfluQ7k6wkeca5blRK\nuSHJDUly+eWX5+TJk5/FcJvr1KlTO97LPX99Or1BLpj3B3tl91wBzs1cgdmYK3B+5gnM5jDOlc86\nSKq1/kqSlFKeUGt978MdQK31NUleU0p5UZL/Lck3nuOaG5PcmCTHjx+vJ06ceLhfthFOnjyZ6ffy\nI7f9tzzq6HJOnLh+cYOCBto9V4BzM1dgNuYKnJ95ArM5jHPloSxtG/ut87x+T5Krpo6vHJ17IK9P\n8vyHMZ4DT48kAAAAoMketCKplPLqB3opySPOc++3J7m2lHJNhgHSC5K8aNf9r621fmB0+PeTfCCH\nmB5JAAAAQJOdL7X4piTflWTzHK+98ME+sdbaK6W8NMlbkrST/HSt9bZSyquS3FJrvSnJS0spz0yy\nleRTOceytsOk0+1lfVVFEgAAANBM5wuS3p7kPbXWP9r9Qinl+85381rrzUlu3nXulVPPv2O2YR4O\nG10VSQAAAEBznS+1+PokZ871Qq31mr0fzuG11R+k2xvokQQAAAA01vmabR+rtXb2ZSSHXKfbT5Ks\nrapIAgAAAJrpfEHSr4yflFLeNOexHGqdbi9JVCQBAAAAjXW+IKlMPX/sPAdy2G1sqkgCAAAAmu18\nQVJ9gOfsMRVJAAAAQNOdr/zlS0op92dYmXR09Dyj41prvXiuoztEJhVJdm0DAAAAGupBU4taq/KY\nfTKpSFr1LQcAAACa6XxL29gnG+Nd2yxtAwAAABpKkNQQp0cVSZa2AQAAAE0lSGqIcY+kdUESAAAA\n0FCCpIYY90g6amkbAAAA0FCCpIbY6Paz0m5lZcmPBAAAAGgmqUVDdDZ7WbNjGwAAANBggqSG2Oj2\n9UcCAAAAGk2Q1BCdbi9r+iMBAAAADSZIaoiNzX7WVlUkAQAAAM0lSGqITreXdRVJAAAAQIMJkhpi\nY7OfNT0RtXULAAAO/ElEQVSSAAAAgAYTJDVEp9vLul3bAAAAgAYTJDXERrev2TYAAADQaIKkhjjd\ntbQNAAAAaDZBUgPUWrOh2TYAAADQcIKkBjizNUitydqqiiQAAACguQRJDbDR7SWJiiQAAACg0QRJ\nDdDZ7CeJHkkAAABAowmSGmBSkbSqIgkAAABoLkFSA3RGQZKKJAAAAKDJBEkNsDFa2qYiCQAAAGgy\nQVIDqEgCAAAADgJBUgNMKpIESQAAAECDCZIaYFKRZGkbAAAA0GCCpAbY6KpIAgAAAJpPkNQAnc1e\nSkmOLPtxAAAAAM0luWiATrefteV2SimLHgoAAADAAxIkNcBGt5+1VcvaAAAAgGYTJDVAp9vL+opG\n2wAAAECzCZIaYGOznzWNtgEAAICGEyQ1QKfby/qqiiQAAACg2QRJDbDRVZEEAAAANJ8gqQE6myqS\nAAAAgOYTJDVAR0USAAAAcAAIkhpgw65tAAAAwAEgSGqAzmY/a6sqkgAAAIBmEyQtWLc3SLc/UJEE\nAAAANJ4gacFOd/tJokcSAAAA0HiCpAXrbPWSJGsqkgAAAICGEyQt2MbmqCJJjyQAAACg4QRJC9bp\nDiuS9EgCAAAAmk6QtGCTiiQ9kgAAAICGm2uQVEp5dinlfaWUO0spLz/H699ZSrm9lPLnpZS3llIe\nM8/xNNGkImlVRRIAAADQbHMLkkop7SSvSfKcJNcleWEp5bpdl92a5Hit9YuTvDHJD8xrPE21Ydc2\nAAAA4ICYZ0XS9UnurLXeVWvtJnl9kudNX1Br/b1aa2d0+LYkV85xPI3U2VSRBAAAABwM8yyDeXSS\nD08d353kqQ9y/Tcn+Y1zvVBKuSHJDUly+eWX5+TJk3s0xMU6depU/uxD70uSvPNP35b15bLgEUEz\nnTp16oKZ9zBP5grMxlyB8zNPYDaHca40Yj1VKeXFSY4n+cpzvV5rvTHJjUly/PjxeuLEif0b3Byd\nPHkyn3f00cl7359nPeMrs9zW+xzO5eTJk7lQ5j3Mk7kCszFX4PzME5jNYZwr8wyS7kly1dTxlaNz\nO5RSnpnke5N8Za11c47jaaSNbj8rSy0hEgAAANB480wv3p7k2lLKNaWUlSQvSHLT9AWllCcl+ckk\nz621/s0cx9JYnW4v6yv6IwEAAADNN7cgqdbaS/LSJG9JckeSN9RabyulvKqU8tzRZT+Y5FiSXy6l\nvKuUctMD3O6CtbHZt2MbAAAAcCDMNcGotd6c5OZd51459fyZ8/z6B0Gn27NjGwAAAHAgaMyzYJ1u\nP0dVJAEAAAAHgCBpwfRIAgAAAA4KQdKC6ZEEAAAAHBSCpAXTIwkAAAA4KARJC7bRVZEEAAAAHAyC\npAXrbOqRBAAAABwMgqQFGtSazlY/a6sqkgAAAIDmEyQt0FY/qTUqkgAAAIADQZC0QGf6w0cVSQAA\nAMBBIEhaoM1+TaIiCQAAADgYBEkLdKY3DJLs2gYAAAAcBIKkBdocLW1bX1WRBAAAADSfIGmBVCQB\nAAAAB4kgaYHGFUlreiQBAAAAB4AgaYG2m22rSAIAAACaT5C0QGfGFUl6JAEAAAAHgCBpgTZ7KpIA\nAACAg0OQtEBn+kkpyZFlPwYAAACg+SQYC7TZq1lfWUopZdFDAQAAADgvQdICnenbsQ0AAAA4OARJ\nC7TZr1lf1R8JAAAAOBgESQt0pqciCQAAADg4BEkLtNmvdmwDAAAADgxB0gKd6SdrqyqSAAAAgINB\nkLRA413bAAAAAA4CQdICbdq1DQAAADhABEkLdKZfBUkAAADAgSFIWqDNXrK2amkbAAAAcDAIkhak\n2xukV5N1FUkAAADAASFIWpDT3X6SZE2zbQAAAOCAECQtyEa3lyRZX1WRBAAAABwMgqQF6YyCJBVJ\nAAAAwEEhSFqQjc3h0jYVSQAAAMBBIUhakA0VSQAAAMABI0hakM64IkmQBAAAABwQgqQFmVQkWdoG\nAAAAHBCCpAXpdFUkAQAAAAeLIGlBNjZVJAEAAAAHiyBpQU6PKpLWlgVJAAAAwMEgSFqQjW4/S61k\nqe1HAAAAABwMUowF6XR7OaIYCQAAADhABEkLsrHZz2q7LHoYAAAAADMTJC1Ip9vLERu2AQAAAAeI\nIGlBNroqkgAAAICDRZC0IJ1NFUkAAADAwSJIWhAVSQAAAMBBI0haELu2AQAAAAeNIGlBNjb7WV1S\nkQQAAAAcHIKkBVGRBAAAABw0cw2SSinPLqW8r5RyZynl5ed4/StKKe8spfRKKV8/z7E0zZv+ly/N\n1zxmedHDAAAAAJjZ3IKkUko7yWuSPCfJdUleWEq5btdlf5XkJUl+cV7jaKonXnFxLj2qIAwAAAA4\nOOa5Af31Se6std6VJKWU1yd5XpLbxxfUWj80em0wx3EAAAAAsAfmGSQ9OsmHp47vTvLUh3KjUsoN\nSW5IkssvvzwnT5582INrglOnTl0w7wXmyVyB2ZgrMBtzBc7PPIHZHMa5Ms8gac/UWm9McmOSHD9+\nvJ44cWKxA9ojJ0+ezIXyXmCezBWYjbkCszFX4PzME5jNYZwr82zSc0+Sq6aOrxydAwAAAOAAmmeQ\n9PYk15ZSrimlrCR5QZKb5vj1AAAAAJijuQVJtdZekpcmeUuSO5K8odZ6WynlVaWU5yZJKeXvlVLu\nTvKPk/xkKeW2eY0HAAAAgIdnrj2Saq03J7l517lXTj1/e4ZL3gAAAABouHkubQMAAADgAiJIAgAA\nAGAmgiQAAAAAZlJqrYsew2ellPLxJH+56HHskcuS3LvoQcABYK7AbMwVmI25AudnnsBsLqS58pha\n66POd9GBC5IuJKWUW2qtxxc9Dmg6cwVmY67AbMwVOD/zBGZzGOeKpW0AAAAAzESQBAAAAMBMBEmL\ndeOiBwAHhLkCszFXYDbmCpyfeQKzOXRzRY8kAAAAAGaiIgkAAACAmQiSAAAAAJiJIGmPlFKeXUp5\nXynlzlLKy8/x+mop5b+MXv+TUsrVU6+9YnT+faWUr531nnAQ7fVcKaVcVUr5vVLK7aWU20op37F/\n7wbmZx5/r4xea5dSbi2l/Pr83wXM35z+DfaIUsobSynvLaXcUUp5+v68G5ifOc2VfzX699d7Sim/\nVEo5sj/vBubnoc6VUsqlo99LTpVSfmzX5zyllPLu0ee8upRS9ufdzIcgaQ+UUtpJXpPkOUmuS/LC\nUsp1uy775iSfqrU+LskPJ/n+0edel+QFSb4wybOT/PjoH/mz3BMOlHnMlSS9JN9Va70uydOSfJu5\nwkE3p7ky9h1J7pjvO4D9Mce58qNJfrPW+oQkXxJzhgNuTr+vPDrJy5Icr7V+UZL26Do4sB7OXEly\nJsn/nuS7z3Hrn0jyPyW5dvTx7L0f/f4RJO2N65PcWWu9q9baTfL6JM/bdc3zkvzc6Pkbk3z1KIV8\nXpLX11o3a60fTHLn6H6z3BMOmj2fK7XWj9Za35kktdbPZPiP/Ufvw3uBeZrH3ysppVyZ5O8nee0+\nvAfYD3s+V0oplyT5iiQ/lSS11m6t9b59eC8wT3P5eyXJUpKjpZSlJGtJPjLn9wHz9pDnSq11o9b6\nhxkGShOllCuSXFxrfVsd7nb280meP9d3MWeCpL3x6CQfnjq+O2f/Iju5ptbaS/LpJJc+yOfOck84\naOYxVyZGZaVPSvInezhmWIR5zZUfSfK/Jhns/ZBhIeYxV65J8vEkPzNaBvraUsr6fIYP+2bP50qt\n9Z4k/2eSv0ry0SSfrrX+1lxGD/vn4cyVB7vn3ee554EiSAIuCKWUY0nelORf1lrvX/R4oGlKKf99\nkr+ptb5j0WOBhltK8uQkP1FrfVKSjSR6VcIupZRHZliZcU2Sz0uyXkp58WJHBewHQdLeuCfJVVPH\nV47OnfOaUennJUk+8SCfO8s94aCZx1xJKWU5wxDpdbXWN89l5LC/5jFX/rskzy2lfCjDMu1nlFL+\n8zwGD/toHnPl7iR311rH1a1vzDBYgoNsHnPlmUk+WGv9eK11K8mbk3zpXEYP++fhzJUHu+eV57nn\ngSJI2htvT3JtKeWaUspKhk3mbtp1zU1JvnH0/OuT/O5ofeRNSV4w6vx+TYaNt/50xnvCQbPnc2W0\ndv+nktxRa/2hfXkXMH97Pldqra+otV5Za716dL/frbX6L8ccdPOYKx9L8uFSyuNHn/PVSW6f9xuB\nOZvH7yt/leRppZS10b/Hvjoa03PwPZy5ck611o8mub+U8rTRXPmGJL+690PfP0uLHsCFoNbaK6W8\nNMlbMtyt4KdrrbeVUl6V5JZa600Z/qL7C6WUO5N8MqMdDUbXvSHDf6D0knxbrbWfJOe6536/N9hL\n85grpZQvS/LPkry7lPKu0Zf6N7XWm/f33cHemdffK3ChmeNc+fYkrxv9EnFXkm/a1zcGe2xOc+VP\nSilvTPLO0flbk9y43+8N9tLDmStJMqr8vjjJSinl+UmeVWu9Pcm/SPKzSY4m+Y3Rx4FVHiQ4AwAA\nAIAJS9sAAAAAmIkgCQAAAICZCJIAAAAAmIkgCQAAAICZCJIAAAAAmIkgCQDgYSilfKiUctnDvQYA\n4CAQJAEAAAAwE0ESAMCMSim/Ukp5RynltlLKDbteu7qU8t5SyutKKXeUUt5YSlmbuuTbSynvLKW8\nu5TyhNHnXF9K+eNSyq2llD8qpTx+X98QAMBnSZAEADC7f15rfUqS40leVkq5dNfrj0/y47XWJya5\nP8m/mHrt3lrrk5P8RJLvHp17b5Ivr7U+Kckrk/z7uY4eAOBhEiQBAMzuZaWUP0vytiRXJbl21+sf\nrrX+t9Hz/5zky6Zee/Po8R1Jrh49vyTJL5dS3pPkh5N84TwGDQCwVwRJAAAzKKWcSPLMJE+vtX5J\nkluTHNl1WX2Q483RYz/J0uj5v03ye7XWL0ryD85xPwCARhEkAQDM5pIkn6q1dkY9jp52jms+v5Ty\n9NHzFyX5wxnuec/o+Uv2ZJQAAHMkSAIAmM1vJlkqpdyR5D9muLxtt/cl+bbRNY/MsB/Sg/mBJP+h\nlHJrtquUAAAaq9S6uwIbAIDPVinl6iS/PlqmBgBwQVKRBAAAAMBMVCQBAAAAMBMVSQAAAADMRJAE\nAAAAwEwESQAAAADMRJAEAAAAwEwESQAAAADM5P8Hpp1urXWfmXsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x130cbe3d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "alphas = np.linspace(0.0, 0.01, 100)\n",
    "scores = [get_score(alpha, y, y_hat_bow) for alpha in alphas]\n",
    "\n",
    "plot(alphas, scores);\n",
    "scatter(alphas[np.argmax(scores)], np.max(scores));\n",
    "grid()\n",
    "ylabel('F1-score')\n",
    "xlabel('alpha')\n",
    "\n",
    "print(np.max(scores))\n",
    "print(alphas[np.argmax(scores)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Hashing Trick\n",
    "\n",
    "Вместо того, чтобы хранить каждый вариант N-граммы отдельно, мы будем получать индекс столбца с признаком, хешируя содержимое.\n",
    "\n",
    "Позволяет задавать произвольное количество столбцов и контролировать максимальное количество признаков соответственно."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 100000)\n",
      "<class 'scipy.sparse.csr.csr_matrix'>\n"
     ]
    }
   ],
   "source": [
    "vectorizer = HashingVectorizer(binary=True, ngram_range=(1, 3), n_features=100000)\n",
    "bow = vectorizer.fit_transform(texts.text)\n",
    "print(bow.shape)\n",
    "print(type(bow))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 280,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "params = {'estimator__C': np.logspace(-5, 5, 11)}\n",
    "\n",
    "clf = OneVsRestClassifier(LogisticRegression())\n",
    "cv = GridSearchCV(clf, params, n_jobs=-1, scoring=make_scorer(f1_score, average='samples'), cv=5)\n",
    "cv.fit(bow, y);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 281,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>params</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.280800</td>\n",
       "      <td>{u'estimator__C': 100000.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.275890</td>\n",
       "      <td>{u'estimator__C': 10000.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.265307</td>\n",
       "      <td>{u'estimator__C': 1000.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.247607</td>\n",
       "      <td>{u'estimator__C': 100.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.200850</td>\n",
       "      <td>{u'estimator__C': 10.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.055257</td>\n",
       "      <td>{u'estimator__C': 1.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>{u'estimator__C': 1e-05}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>{u'estimator__C': 0.0001}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>{u'estimator__C': 0.001}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>{u'estimator__C': 0.01}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>{u'estimator__C': 0.1}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    mean_test_score                       params\n",
       "10         0.280800  {u'estimator__C': 100000.0}\n",
       "9          0.275890   {u'estimator__C': 10000.0}\n",
       "8          0.265307    {u'estimator__C': 1000.0}\n",
       "7          0.247607     {u'estimator__C': 100.0}\n",
       "6          0.200850      {u'estimator__C': 10.0}\n",
       "5          0.055257       {u'estimator__C': 1.0}\n",
       "0          0.000000     {u'estimator__C': 1e-05}\n",
       "1          0.000000    {u'estimator__C': 0.0001}\n",
       "2          0.000000     {u'estimator__C': 0.001}\n",
       "3          0.000000      {u'estimator__C': 0.01}\n",
       "4          0.000000       {u'estimator__C': 0.1}"
      ]
     },
     "execution_count": 281,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(cv.cv_results_)[['mean_test_score', 'params']].sort_values('mean_test_score', ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 48.1 s, sys: 508 ms, total: 48.6 s\n",
      "Wall time: 48.8 s\n"
     ]
    }
   ],
   "source": [
    "%time y_hat_bow = cross_val_predict(clf, bow, y, method='predict_proba')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Выбор порога для HashingTrick\n",
    "\n",
    "* Сократив количество признаков до $100000$ мы получили $F1=0.525578$\n",
    "* Качество уменьшилось не сильно, но при этом время затраченное на ```cross_val_predict``` сократилось более чем в 2 раза до ```48.8 s```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.525578571429\n",
      "0.00222222222222\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABIQAAAEyCAYAAACLeQv5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuUpGldJ/jvE5GXuveFLuwrXS2i2IAKFA3oHtxRdqSd\nOYAj7oCrqOsO41F2XGd2R3B3PLuM7ll115nZGWZWVh2VUdFlGOlVhHHV2aNnEbobGKEb0Ba66St0\n05fKrKqIjMuzf0REZmR2NZV0RWREVH4+xzzvJd5844kqXyrz28/v95RaawAAAADYPxqzHgAAAAAA\ne0sgBAAAALDPCIQAAAAA9hmBEAAAAMA+IxACAAAA2GcEQgAAAAD7jEAIAAAAYJ8RCAEAAADsMwIh\nAAAAgH1maVZvfMUVV9QTJ07M6u0BAAAALjq33377I7XW4+e7bmaB0IkTJ3LbbbfN6u0BAAAALjql\nlHt2c52SMQAAAIB9RiAEAAAAsM8IhAAAAAD2GYEQAAAAwD4jEAIAAADYZwRCAAAAAPuMQAgAAABg\nnxEIAQAAAOwzAiEAAACAfUYgBAAAALDPLM16AABP1+989P783Ac+nQceP5urLz2Y/+7bviavfeE1\nsx4WAADA3BMIAQvpdz56f976no/nbKeXJLn/8bN563s+niRCIQAAgPMQCMGCq7XmzEYvT5zt5FSr\nk063pl9HX4PX+zWb5+pwf6SkpJSkDA62HZcy2K/D+9QMv3fwf4PzGdxz837lyfdslPKU9xvdY+c9\nk6Rfk16/n26vptev6fa3tj/1u3duhkEjZzu9/I+33JEzG700Ru87tm02Sg6tLOXI6vDrwFIOrzZz\ndHU5B5YbKaVM8W8KAABgfgiEYA50e/2stbo51erk1NnRtjN2rpPHz3byxI6vU8Ntp1fP/yb7xONn\nO/mJf/fxL/v7GiWbQdHh1aUsNxtpNAbhVqMMwqxRyDQKmEYhU6OUNBslzVLSGG6bjdF+Ns+Nh1PZ\nvO+TA7PBUbZCteF+htcMvjsZ/a3XYdD3VCFds1FyYLmRA0vNHFhu5sByI6vLw/2lxvBcM6tLjSw3\nG1luliw1G1lqlCw3G1lqlq3zjcFWeAYAAItNIARPQ69fs97uZq3VyXq7m/VWN2ujbaub0+1uzmz0\ncqbTzdmNXk63eznbGZ7b6A3ObXRzpt3LqVYnZzZ6X/L9SkmOHVjOJQe3vq6+5GAuObT93LEDy1ld\nGgYZw+Bi50yZzTAjYwHCeWbrbM74eYoZRGU4xm2hxI57Ds7tuF/ylLOSRvdcajTSbJQsNUuWGiXN\nxiCoeN3/8f/l86faT/qzuvLYgbz3zd+0OUOq39+aFdXt15zd6GWt3cnpdi/r7U7W272sD//O1kdf\nrW66/TqcXTWcaZVsHfcH9+v1azrD2Uuj4/H9wQynrXOjz98f/lkk22dxjYKdZOv9RvtJxl4bXHeu\n8KgMw6WMHXd7/bS6/fT6kw8Od75vY3hi9Pe3utTMoZXR11IOrmwdH1xe2trf3C7l0PLWucOrSzm4\nPPb9y82sLjeyumRGFwAAXAiBEPtep9fPY6c38sXTG3l0tF1vb+2PbU+dHQRA5wtwRlaWGoNfZJeb\nObS6NPwluJnjR1fzrJVDObzSzLEDyzl2cDnHDizl2MHlHD2wtT86f3hlKY2GX37HvfXmr93WQyhJ\nDi4385abn5uvOHZghiObX51eP61OL63OYNvubu23Ov20u710ejWdXj/dfj+dXk23V8f2++n2B6+P\nwr9sBmZPnqXU79e0u/3NEPTMxuDZefT0Ru57rJcz7W7OdAYh6Ua3/2V/npWlQTC0ujSc9TTcX11u\nZKU5WERzFAaeu4RycLzcHH7vcBbV+HZ1bHtwuZkjq4Ng6vDY9vDq4Bk9NAywhFUAACwCgRAXtX6/\n5uH1dh54/GwefKKVBx4/mwceb+XBJ84O9p9o5ZH19rbympFSkssOreTyw4Ov5zzzSC45uLzZe+bI\n6lKOHVje3D9yYClHN/vSDGY5LA1/KWXyRo2jrTK2e4Oyr0aOzmFe1u31c7YzCo6GM+l2zKo7MwyV\nNnr9tDv9tLq9tDv9tLuDMKvd7Q+PB/slg3K55cZoZtx42d/WrKZuv26GYo+f6Wzer9Xpp93ppdXt\nf9mB1ai8bmk4u605LLVrjsrwGoOyvJVmycqwVG98u9rc2l9ZamSUB2+fAbZzdtjWLL3NfmH9uv14\nbBZaszGcdTecfTcab3PHbLzRLLfe2Cy4c82KazYaYyHdcOzNxjCga26+Nvqcq9s+d9m8frRtDj/0\nzs80XhI52o7KHAEA2D2BEAuv1enl3kfP5LOPnM49XzyTz37xdO5+5HQ+9+iZfP5U60n9dQ6tNHPV\nJQdy9aUH89wrj+XKSw7kiqOrecYw+BltLz20svkLCfPptS+8RgB0kVhqNnK02cjRA8uzHso5jWY7\nDWYIdnO6PQinTm/0crq9VSZ6eqOb1kYvnf6opLC/WVrYHe33a3r9fja6g9c7vUHgdLrdTbs7PO71\n0+nWbAxf628rJ6xjM7S2H4/K9sb7VZ1rW8fCnO5Yw/YvR2NHD61ubzDeWWmUbAVqO0O24f5gvNtL\nHBuNPCkwHAVs472+RqWt28siy/Dvpo7NRsuTjpNkddiv6+Cwj9eod9eor9fBsV5eoxBtVB65Mpr9\ntrT1+ZYajbFS4K0+Z2anAQC7JRBibnV7/ZxqdZ/UQPmhJ1q5+4unB1+PnMkDT5zdNsPn0kPLOfGM\nw3nx9Zfl6ksP5uph+HPVJQdzzaUHc+zgkh+YgS9Lo1FycNjXKFmd9XCmYhQSba7m1xvMCmoOQ5Nm\nY6t5erNx7sbitQ5CofZwVtVGd+d+bzPk6vTq4Hyvl063pt3rp9MdhWH9dPp1W8A1Cmk2G6uPzZIa\nD9Y2Q7XRvcbO7ywhHIV140FOr473PBtv2l63zVYa9Vrb2XR+M2waC5pSB4Fiq9PL2W1lm9MJ0HaO\naVtz+OHstKVmycpwu9QYhGbLS4Nz47PTxmdtLQ+3g2CrkYMrW6HWweXB83FgqZmDK4NrVpqNzXGM\n/n9mFMqNetqNh3QAwN4SCDEz/X7NX3xhLbd+9tHcfs9jefCJVp4Yrqz1xLBXz1MZhT4vOXFZTlxx\nbW644nCuf8bhnHjGoVx6aGUPPwXAxaGUYfP25oXdYzCT5QJuso+MZp5tBUXD0sfxAK27VRa5ud8Z\n9PMahVT9/lYvr61eWYPXNmeo7egFttHrD/Z7g1lrneF911rdzTBtYyxc2wrwphNijcorlxvjKxs2\nNhcUWB6WEY5KFLcFl8Mgs1+3ZuIl526yP94IfxRUbQVfzc2SzZ2B2HKzsX1lyLJ9pcjxGXjj5ZGr\nS81tpZRb5ZTNrZB1GB5uW7WysT0w27k4xOZMt/FZYqMPeh6lZFDOqd8ZwL4nEGLPbHT7+fj9T+TW\nux/NrZ99NLfd81ieONtJknzFsdVcf/nhXHvZoe2rZh1c2nZ8ycHlHD+6KvQBYOGNzzy7bNaD2aU6\nnO10dmMrxNrcbgx7gXV6w9LJ/mZYNeo5VWvS21yxcfBat7+9af2omf1Gd7Dt9rbKLxuj3leNrV5X\nzUY2e16NwpTBWHf2nNq++mW/Jr3eVmlmezMIG5SEPn52e+lmsrNJ/djqk6PZZ8OwbWe5+rzaDKpG\n5YrD0sZRcLXUaKTRKGnuKBEdlFsOzjca2/uOja8KOuqdNn48uv+BsTLJUenkaJGAg8vNHDu4LLQC\nmDKBEFOz3u7mo597LLd+9tF8+O5H87F7H0+rM/iB6iuPH87Nz78yLzlxeW664fJce9lB/+ADwJwr\npWz+Ir8oIdYsjIKhzab3nf5mQ/zBtjcIxvrDgGyseXt/rIl7f3hNfzPMevJKiTv7VZ3PKNQbH9tg\n5llvW5P+VqeXjW43veEstM3xbI4t2xrLd3qD/mjjs7e6/f6ux3UuK83GcNXVwX8gPHZg+38wPHZg\nOQeWm08qc9yciTXWR2x1ubEtgBr1GgPYzwRCTMwDj5/Nbfc8ltvvfjS33v1YPvXQqfTroJfBjVcf\nyxtuelZuOnF5Tp64PMePXpw9OAAAGo2SA41BcJbMZ7P8vdIf6002CsNGgVOrs7WqY6vT21w98kyn\nl7XWqIdkN6fOdnKq1cljZzZyzxdPD863ul92M/ydmo2yLSAa9b5aXhr01lpujnpwDVaFXGoMemkt\nD0sNDwx7Zx1cbubQsKfWoZWtnlqbvbU2Z2CNN45vWB0RmDmBEE9Lr1/zyQdP5fZ7HtsMgR54opVk\nsIrXN1x3ad78174qLz5xeV70rEvnduUgAACmp9EoWRmW8R1MMzk4mZ8Ja605szEoVxzveTXeyH50\nfrTf7vQ3Q6dREDW+3erR1c/GsOdWp9fP6Y3e5n53rMxwVDL5dEsER4HUKCwalduNyvAaw95uzbHe\nUs1hT63x0GlnELW1omEzzcZgZt9ogYDxhu6jHlSbZYA7SgPPdb7WpNPf+rPojMo9h4sBjP6cev3t\nfc0Gf2djDfpHq2UmY6WGW2WHo75hy6PzzbK5kMDIeHHBtv1s7xc22t9apGCwbTbKWFgnoGN/Egix\na2utTv7oU1/I+z/xUP7kLx/ZbPp85bEDefGJy/J3rr8sJ6+/PF971VH/gwoAwNSUUnJ4dSmHV2f/\n60ynNwyHhr21znZ6g7BqY7AdL8VrjzWP31ay1+1tru7Y7ddtM6t2fq11unl4rT14r7F+XovSu2pe\nLe2YMbY6tl1plm0liMvDBvTLY43nl5eGDeaTbcHToLl92RZKjcKuUQP9p95v7GhM39wsh9zq9XXu\nlT9hN3b1v6CllFcl+WdJmkl+sdb6v+x4/fuT/FyS+4en/kWt9RcnOE5m5LHTG/mDOz+f99/xUP70\nLx/JRq+fZx5dzau/4eq89IbL8+LrL8s1l+r/AwDA/jT6xX3WM+J3BlOtTv9LNnQfnB/2rhrrFTXe\ny2rn+UbZCjJGKwCOVgQczeYZze4ZzEBKsnPGTrYCkySD5vFjfadGKyL2+nVs1tHOsGvruNbtZ7fN\nRMpWr62aQU+u0bluv27O9GqPbbdmjW2d63T7ae1YifFcqzBuvdf2RvbT1CjZCqWeIlwaL3sc/DkN\n+46N/Vn163jj/cF9lxpbM7Saja1ZXM3hbLalYWP/UaP50aqI21dG3Go+PyivXMqhlWYOrS7l0HIz\nh1YH5w4PF1k4tLK0bYXFzZld51hpkQt33kColNJM8vYk/1mS+5LcWkq5pdZ6545Lf6vW+uYpjJE9\n9vlTrfz7Ox7K73/ioXzos4+m16+59rKDeePLr8/NL7gyL7zusjQaHkAAAJgX8xJMcW6jpvCjEK7T\nG5Xa9Tf3RyssDkrxtkogx0shB/u97a8NA6pOb7A6Yqc3KOHrDEspN8ZK/JJBsJKSNBpJSWMzbBll\nLKWUQWjWG4SB7W5vM7Tr9eu2EG+r4fyTV16sw+CxX7dmvE3SeInhKAA7V/+v8RLInSsl7jz/09/x\n/H21ovVuZgjdlOSuWutnkqSU8q4kr0myMxBigT16eiPv/dj9+d0/fzAf+dxjqXWwEtgPffNX5ubn\nX5XnXX1MCgsAAPA0DHo5JUnJUpLVpST7bJ2djW4/Zza6ObPR29yebvdyttMdbIfne08RLm1babE/\nFqzt6PHV3TxXhz2ttma5bXSHM+eeYgXFSYdW8243gdA1Se4dO74vyUvPcd13llJekeQvkvxYrfXe\nnReUUt6U5E1J8qxnPevLHy0T1en18x8+/XDeffu9+aNPfSGdXs3XXnUsP/bKr87Nz78yz/mKo7Me\nIgAAABeBQf+jlVx6aNYjYWRSXdj+7yS/WWttl1L+bpJfTfItOy+qtb4jyTuS5OTJk/srepsjdz5w\nKu++/b6892P354unN3LFkZV8/zeeyHe++No898pjsx4eAAAAMGW7CYTuT3Ld2PG12WoenSSptX5x\n7PAXk/zshQ+NSfriejvv/dgDefft9+XOB09luVnyyq/9irzuxdfmFV99PMtWBQMAAIB9YzeB0K1J\nnlNKuSGDIOj1Sb57/IJSylW11geHh69O8smJjpKn7aEnWvmp37sz7//EQ+n2a15wzSX5n179vLz6\n66/OZYf3T7MsAAAAYMt5A6Faa7eU8uYkH8hg2flfrrXeUUp5W5Lbaq23JPl7pZRXJ+kmeTTJ909x\nzOxCrTW/deu9+enf+2Q6/X6+/xtP5HUnlYQBAAAASal1Nq18Tp48WW+77baZvPfF7t5Hz+St7/l4\n/vSuR/LSGy7Pz3zn1+XEFYdnPSwAAABgykopt9daT57vukk1lWYO9Ps1v/bBu/Mz7/90mo2Sn3rt\n8/PdNz0rjYbl4gEAAIAtAqGLxF89vJ4ff/ef57Z7Hss3f/Xx/M9/6wW55tKDsx4WAAAAMIcEQguu\n2+vn//yTz+af/D9/kYPLzfxv3/X1+VsvuialmBUEAAAAnJtAaIF98sFT+Yfv/vN8/P4n8qrnXZm3\nvfZ5eebRA7MeFgAAADDnBEIL6g8/+fn80L+5PZccXM6//C9elG9/wVWzHhIAAACwIARCC+jDn300\nP/zrH8mNVx3Lv/6Bm3L54ZVZDwkAAABYII1ZD4Avzx0PPJEf/JVbc+1lB4VBAAAAwNMiEFogn33k\ndL7vlz+coweW8s4ffKkwCAAAAHhaBEIL4qEnWvneX/pQ+jV553/10lxtSXkAAADgaRIILYDHz2zk\njb/8oTx2eiO/+gM35dnHj8x6SAAAAMAC01R6zp3Z6OYHfuXW3P3ImfzKf/mSvODaS2Y9JAAAAGDB\nmSE0xza6/fzdd96e/3jv4/nn3/3CfOOzr5j1kAAAAICLgBlCc6rXr/mx3/5Y/uQvH8nPvu7r8m3P\nu3LWQwIAAAAuEmYIzaFaa/7Rez+R3/vzB/MT3/7c/Ocnr5v1kAAAAICLiEBoDv38H/xFfuNDn8sP\nffOz86ZXPHvWwwEAAAAuMgKhOXPLf3wg//yP7srrX3JdfvxVXzPr4QAAAAAXIYHQHKm15hf+37/K\n13zF0fz0d7wgpZRZDwkAAAC4CAmE5shHPvd47njgVL735den2RAGAQAAANMhEJoj7/zg3Tm6upTv\neOE1sx4KAAAAcBETCM2JR9bbed/HH8p3vvjaHF5dmvVwAAAAgIuYQGhO/Nat92aj18/3vOz6WQ8F\nAAAAuMgJhOZAt9fPr//ZPfmmr3pGvuqZR2Y9HAAAAOAiJxCaA3/4qS/kgSda+d6XnZj1UAAAAIB9\nQCA0B975wXty9SUH8sqvfeashwIAAADsAwKhGfurh9fzp3c9ku9+6bOy1PTXAQAAAEyfBGLG3vnB\ne7LcLPnbL3nWrIcCAAAA7BMCoRk63e7m395+X779BVfl+NHVWQ8HAAAA2CcEQjP0Ox+7P2vtbt74\nckvNAwAAAHtHIDQjtda884P35MarjuVFz7ps1sMBAAAA9hGB0Izcevdj+dRDa3njy69PKWXWwwEA\nAAD2EYHQjPzaB+/OsQNLec03XDProQAAAAD7jEBoBr5wqpX3f+KhfNfJ63JwpTnr4QAAAAD7jEBo\nBn7zw/em26/5npdpJg0AAADsPYHQHuv0+vmND9+TV3z18dxwxeFZDwcAAADYhwRCe+wP7vx8Pn+q\nnTeaHQQAAADMiEBoj/3aB+/ONZcezF977jNnPRQAAABgnxII7aG/+Pxa/uwzj+Z7XnZ9mg1LzQMA\nAACzIRDaQ+/84D1ZWWrkb7/kulkPBQAAANjHBEJ7ZK3VyXs+cl/+5tddlcsPr8x6OAAAAMA+JhDa\nI//uo/fn9EYvb3z5iVkPBQAAANjnBEJ75I8/9YU8+/jhfMN1l856KAAAAMA+JxDaI6da3Vx5yYFZ\nDwMAAABAILRX1lvdHFldmvUwAAAAAARCe2Wt1cmR1eVZDwMAAABAILRX1trdHD1ghhAAAAAwewKh\nPVBrzbpACAAAAJgTAqE9cHqjl1qjhxAAAAAwFwRCe2C91U2SHD2ghxAAAAAwewKhPbDe7iRJjigZ\nAwAAAObArgKhUsqrSimfLqXcVUp5y5e47jtLKbWUcnJyQ1x8p0YzhJSMAQAAAHPgvIFQKaWZ5O1J\nbk5yY5I3lFJuPMd1R5P8aJIPTXqQi26rZEwgBAAAAMzebmYI3ZTkrlrrZ2qtG0neleQ157juHyf5\nmSStCY7vorDeHgRCSsYAAACAebCbQOiaJPeOHd83PLeplPKiJNfVWn/vS92olPKmUsptpZTbHn74\n4S97sItqrTXsIaRkDAAAAJgDF9xUupTSSPLzSf7B+a6ttb6j1nqy1nry+PHjF/rWC2PNKmMAAADA\nHNlNIHR/kuvGjq8dnhs5muT5Sf5DKeXuJC9LcovG0ls2S8bMEAIAAADmwG4CoVuTPKeUckMpZSXJ\n65PcMnqx1vpErfWKWuuJWuuJJH+W5NW11tumMuIFtNbq5tBKM81GmfVQAAAAAM4fCNVau0nenOQD\nST6Z5LdrrXeUUt5WSnn1tAd4MVhvda0wBgAAAMyNXaUUtdb3JXnfjnM/+RTX/qcXPqyLy3q7q1wM\nAAAAmBsX3FSa8zvV6uSIhtIAAADAnBAI7YH1djfHlIwBAAAAc0IgtAfWW0rGAAAAgPkhENoDawIh\nAAAAYI4IhPbAerubo3oIAQAAAHNCIDRl/X4drDKmhxAAAAAwJwRCU7a+0U2SHFUyBgAAAMwJgdCU\nrbeGgZAZQgAAAMCcEAhN2Xp7EAgpGQMAAADmhUBoytZanSSxyhgAAAAwNwRCU7a2WTJmlTEAAABg\nPgiEpmxUMqaHEAAAADAvBEJTNpohpGQMAAAAmBcCoSmzyhgAAAAwbwRCU7Y2LBk7vCIQAgAAAOaD\nQGjK1lvdHFldSqNRZj0UAAAAgCQCoalba3WUiwEAAABzRSA0ZevtrobSAAAAwFwRCE3ZerubI2YI\nAQAAAHNEIDRlp1rdHD2wPOthAAAAAGwSCE3ZequTo0rGAAAAgDkiEJoyPYQAAACAeSMQmrK1Vtcq\nYwAAAMBcEQhNUa9fc2ajp6k0AAAAMFcEQlO03u4miZIxAAAAYK4IhKZordVJkhyzyhgAAAAwRwRC\nU7Q5Q0jJGAAAADBHBEJTtN5SMgYAAADMH4HQFK0NAyGrjAEAAADzRCA0RWttgRAAAAAwfwRCU7RV\nMqapNAAAADA/BEJTNFplzAwhAAAAYJ4IhKZovd1NKcmhleashwIAAACwSSA0RWutbo6sLqWUMuuh\nAAAAAGwSCE3RWqubYwf0DwIAAADmi0BoitbbnRxZ1T8IAAAAmC8CoSlab3dzRENpAAAAYM4IhKZo\nrdW1whgAAAAwdwRCU7Q+bCoNAAAAME8EQlO01jZDCAAAAJg/AqEpWmt1ctQqYwAAAMCcEQhNSafX\nT6vTVzIGAAAAzB2B0JScbneTRCAEAAAAzB2B0JSstQaBkB5CAAAAwLwRCE2JQAgAAACYVwKhKVnf\nLBnTVBoAAACYLwKhKVlrdZKYIQQAAADMH4HQlGzOEBIIAQAAAHNGIDQlmz2ErDIGAAAAzJldBUKl\nlFeVUj5dSrmrlPKWc7z+Q6WUj5dSPlZK+dNSyo2TH+pi2WoqrYcQAAAAMF/OGwiVUppJ3p7k5iQ3\nJnnDOQKf36i1vqDW+g1JfjbJz098pAtmvd1Js1FyYNkkLAAAAGC+7CatuCnJXbXWz9RaN5K8K8lr\nxi+otZ4aOzycpE5uiItpvdXNkdWllFJmPRQAAACAbXbT4OaaJPeOHd+X5KU7Lyql/EiSv59kJcm3\nTGR0C2yt1bXCGAAAADCXJlbPVGt9e6312Ul+PMn/cK5rSilvKqXcVkq57eGHH57UW8+ltfZghhAA\nAADAvNlNIHR/kuvGjq8dnnsq70ry2nO9UGt9R631ZK315PHjx3c/ygW0boYQAAAAMKd2EwjdmuQ5\npZQbSikrSV6f5JbxC0opzxk7/BtJ/nJyQ1xMa+2OGUIAAADAXDpvYlFr7ZZS3pzkA0maSX651npH\nKeVtSW6rtd6S5M2llFcm6SR5LMn3TXPQi2C91c1XXnFk1sMAAAAAeJJdTWGptb4vyft2nPvJsf0f\nnfC4Ft56u5sjSsYAAACAOTSxptJst9bq5qiSMQAAAGAOCYSmYKPbT7vb11QaAAAAmEsCoSlYb3eT\nRFNpAAAAYC4JhKZgvTUMhA4sz3gkAAAAAE8mEJqCU61OkigZAwAAAOaSQGgKRiVjmkoDAAAA80gg\nNAVbJWMCIQAAAGD+CISmYK09KhnTQwgAAACYPwKhKdicIaRkDAAAAJhDAqEpWBv1EFIyBgAAAMwh\ngdAUrLW6WW6WrC754wUAAADmj8RiCtZb3RxZXUopZdZDAQAAAHgSgdAUrLe7VhgDAAAA5pZAaArW\nWp0cXbXCGAAAADCfBEJTsNYyQwgAAACYXwKhKVhvd3PUkvMAAADAnBIITcFaq2vJeQAAAGBuCYSm\nQFNpAAAAYJ4JhKZgsOy8ptIAAADAfBIITVir08tGr69kDAAAAJhbAqEJW293k0QgBAAAAMwtgdCE\nrbcGgdARq4wBAAAAc0ogNGFrrdEMIT2EAAAAgPkkEJqwtXYniRlCAAAAwPwSCE3YeksPIQAAAGC+\nCYQmbE0gBAAAAMw5gdCEjVYZUzIGAAAAzCuB0IRtBkJmCAEAAABzSiA0YadanawsNbK61Jz1UAAA\nAADOSSA0Yeutbo4qFwMAAADmmEBowtbbXeViAAAAwFwTCE3YWqtrhTEAAABgrgmEJmy91bXCGAAA\nADDXBEITttbu5sjq8qyHAQAAAPCUBEITttbq5JiSMQAAAGCOCYQmTFNpAAAAYN4JhCao1qqHEAAA\nADD3BEIT1Or00+3XHD2ghxAAAAAwvwRCE7TW7iSJkjEAAABgrgmEJmi91U2SHFUyBgAAAMwxgdAE\nrY0CITOEAAAAgDkmEJqg9fYgENJUGgAAAJhnAqEJGs0Q0kMIAAAAmGcCoQkazRA6ZpUxAAAAYI4J\nhCZorTVcZUzJGAAAADDHBEITtK5kDAAAAFgAAqEJWm93c2C5keWmP1YAAABgfkkuJuhUq5sjq/oH\nAQAAAPM5kFegAAALOElEQVRNIDRB6+1ujioXAwAAAOacQGiC1lsdgRAAAAAw93YVCJVSXlVK+XQp\n5a5SylvO8frfL6XcWUr581LKH5ZSrp/8UOffWqtrhTEAAABg7p03ECqlNJO8PcnNSW5M8oZSyo07\nLvtokpO11q9L8u4kPzvpgS6C9bZACAAAAJh/u5khdFOSu2qtn6m1biR5V5LXjF9Qa/3jWuuZ4eGf\nJbl2ssNcDGutbo4e0FQaAAAAmG+7CYSuSXLv2PF9w3NP5QeT/P65XiilvKmUclsp5baHH35496Nc\nEGt6CAEAAAALYKJNpUsp35PkZJKfO9frtdZ31FpP1lpPHj9+fJJvPXO1ViVjAAAAwELYTXpxf5Lr\nxo6vHZ7bppTyyiT/fZJvrrW2JzO8xXG200u/xgwhAAAAYO7tZobQrUmeU0q5oZSykuT1SW4Zv6CU\n8sIkv5Dk1bXWL0x+mPNvrdVNkhwRCAEAAABz7ryBUK21m+TNST6Q5JNJfrvWekcp5W2llFcPL/u5\nJEeS/F+llI+VUm55ittdtDYDISVjAAAAwJzbVXpRa31fkvftOPeTY/uvnPC4Fs56exAIHbPKGAAA\nADDnJtpUej9ba3WSKBkDAAAA5p9AaELWlYwBAAAAC0IgNCFrw5Ixq4wBAAAA804gNCGjptJHV/UQ\nAgAAAOabQGhCRiVjh1ebMx4JAAAAwJcmEJqQ9XYnh1aaWWr6IwUAAADmm/RiQtZaXQ2lAQAAgIUg\nEJqQtXbXkvMAAADAQhAITch6q5ujBzSUBgAAAOafQGhC1lqdHFUyBgAAACwAgdCErLf1EAIAAAAW\ng0BoQgYlYwIhAAAAYP4JhCZkraWpNAAAALAYBEIT0O/XrG909RACAAAAFoJAaALOdHqpNVYZAwAA\nABaCQGgC1lqdJFEyBgAAACwEgdAErLe6SWKVMQAAAGAhCIQmYK09CISsMgYAAAAsAoHQBKy1BEIA\nAADA4hAITcBWyZim0gAAAMD8EwhNwHp70FTaDCEAAABgEQiEJmBUMmaVMQAAAGARCIQmYBQIHV4R\nCAEAAADzTyA0Aevtbo6sLqXZKLMeCgAAAMB5CYQmYK3VyZFVs4MAAACAxSAQmoD1dlf/IAAAAGBh\nCIQmYK3VtcIYAAAAsDAEQhMw6iEEAAAAsAgEQhNghhAAAACwSARCE7De6ubo6vKshwEAAACwKwKh\nCdBUGgAAAFgkAqEL1OtXPYQAAACAhSIQukCnN7pJoocQAAAAsDAEQhdovSUQAgAAABaLQOgCrQ0D\noSOaSgMAAAALQiB0gdbbnSRmCAEAAACLQyB0gTZnCAmEAAAAgAUhELpAo0DoqFXGAAAAgAUhELpA\n6+1RU2k9hAAAAIDFIBC6QOtKxgAAAIAFIxC6QGutTkpJDi03Zz0UAAAAgF0RCF2gtXY3R1aX0miU\nWQ8FAAAAYFcEQhdovdXVUBoAAABYKAKhC7TW6uofBAAAACwUgdAFWh+WjAEAAAAsCoHQBVprdy05\nDwAAACwUgdAFWmt1lIwBAAAAC0UgdIE0lQYAAAAWjUDoAq23uzlqhhAAAACwQHYVCJVSXlVK+XQp\n5a5SylvO8forSikfKaV0Symvm/ww59d7fvgb8wPfdMOshwEAAACwa+cNhEopzSRvT3JzkhuTvKGU\ncuOOyz6X5PuT/MakBzjvnnvlsVx96cFZDwMAAABg13ZT63RTkrtqrZ9JklLKu5K8JsmdowtqrXcP\nX+tPYYwAAAAATNBuSsauSXLv2PF9w3NftlLKm0opt5VSbnv44Yefzi0AAAAAuEB72lS61vqOWuvJ\nWuvJ48eP7+VbAwAAADC0m0Do/iTXjR1fOzwHAAAAwALaTSB0a5LnlFJuKKWsJHl9klumOywAAAAA\npuW8gVCttZvkzUk+kOSTSX671npHKeVtpZRXJ0kp5SWllPuSfFeSXyil3DHNQQMAAADw9O1mlbHU\nWt+X5H07zv3k2P6tGZSSAQAAADDn9rSpNAAAAACzJxACAAAA2GcEQgAAAAD7TKm1zuaNS3k4yT0z\nefPJuyLJI7MeBCwAzwrsjmcFdsezAufnOYHduZieletrrcfPd9HMAqGLSSnltlrryVmPA+adZwV2\nx7MCu+NZgfPznMDu7MdnRckYAAAAwD4jEAIAAADYZwRCk/GOWQ8AFoRnBXbHswK741mB8/OcwO7s\nu2dFDyEAAACAfcYMIQAAAIB9RiAEAAAAsM8IhHYopbyqlPLpUspdpZS3nOP11VLKbw1f/1Ap5cTY\na28dnv90KeXbdntPWESTflZKKdeVUv64lHJnKeWOUsqP7t2ngemZxr8rw9eapZSPllJ+d/qfAqZv\nSj+DXVpKeXcp5VOllE+WUl6+N58GpmdKz8qPDX/++kQp5TdLKQf25tPA9DzdZ6WU8ozh7yXrpZR/\nseN7XlxK+fjwe/73UkrZm08zHQKhMaWUZpK3J7k5yY1J3lBKuXHHZT+Y5LFa61cl+SdJfmb4vTcm\neX2S5yV5VZJ/OfxhfTf3hIUyjWclSTfJP6i13pjkZUl+xLPCopvSszLyo0k+Od1PAHtjis/KP0vy\n/lrrc5N8fTwzLLgp/b5yTZK/l+RkrfX5SZrD62BhXcizkqSV5B8l+W/Pcet/leTvJHnO8OtVkx/9\n3hEIbXdTkrtqrZ+ptW4keVeS1+y45jVJfnW4/+4k3zpMBV+T5F211nat9bNJ7hrebzf3hEUz8Wel\n1vpgrfUjSVJrXcvgh/Zr9uCzwDRN49+VlFKuTfI3kvziHnwG2AsTf1ZKKZckeUWSX0qSWutGrfXx\nPfgsME1T+XclyVKSg6WUpSSHkjww5c8B0/a0n5Va6+la659mEAxtKqVcleRYrfXP6mB1rl9L8tqp\nfoopEwhtd02Se8eO78uTfyHdvKbW2k3yRJJnfInv3c09YdFM41nZNJyu+cIkH5rgmGEWpvWs/NMk\n/zBJf/JDhpmYxrNyQ5KHk/zrYXnlL5ZSDk9n+LBnJv6s1FrvT/K/JvlckgeTPFFr/fdTGT3snQt5\nVr7UPe87zz0XikAImCullCNJ/m2S/6bWemrW44F5U0r5m0m+UGu9fdZjgTm3lORFSf5VrfWFSU4n\n0csRdiilXJbBTIkbklyd5HAp5XtmOypgLwiEtrs/yXVjx9cOz53zmuGUykuSfPFLfO9u7gmLZhrP\nSkopyxmEQb9ea33PVEYOe2saz8o3JXl1KeXuDKY/f0sp5d9MY/Cwh6bxrNyX5L5a62i26bszCIhg\nkU3jWXllks/WWh+utXaSvCfJN05l9LB3LuRZ+VL3vPY891woAqHtbk3ynFLKDaWUlQyaqd2y45pb\nknzfcP91Sf5oWD94S5LXDzuV35BBg6kP7/KesGgm/qwMa9t/Kckna60/vyefAqZv4s9KrfWttdZr\na60nhvf7o1qr/5LLopvGs/JQkntLKV8z/J5vTXLntD8ITNk0fl/5XJKXlVIODX8e+9ZowM7iu5Bn\n5ZxqrQ8mOVVKednwWXljkvdOfuh7Z2nWA5gntdZuKeXNST6QQXf9X6613lFKeVuS22qtt2TwC+s7\nSyl3JXk0ww78w+t+O4MfNLpJfqTW2kuSc91zrz8bTNI0npVSyn+S5HuTfLyU8rHhW/1ErfV9e/vp\nYHKm9e8KXGym+Kz810l+ffjLwGeS/MCefjCYsCk9Kx8qpbw7yUeG5z+a5B17/dlgki7kWUmS4Uzs\nY0lWSimvTfLXa613JvnhJL+S5GCS3x9+LazyJQIwAAAAAC5CSsYAAAAA9hmBEAAAAMA+IxACAAAA\n2GcEQgAAAAD7jEAIAAAAYJ8RCAEAAADsMwIhAAAAgH3m/wcZug25mfUh8wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x110e90a10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "alphas = np.linspace(0.0, 0.01, 100)\n",
    "scores = [get_score(alpha, y, y_hat_bow) for alpha in alphas]\n",
    "\n",
    "plot(alphas, scores);\n",
    "scatter(alphas[np.argmax(scores)], np.max(scores));\n",
    "print(np.max(scores))\n",
    "print(alphas[np.argmax(scores)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## VW\n",
    "\n",
    "* Разработка Yahoo и потом Microsoft\n",
    "* Библиотека и __CLI__ программа, позволяющая строить линейные модели\n",
    "* Способна обрабатывать миллиарды объектов с сотнями тысяч признаков"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Установка VW\n",
    "\n",
    "* Ubuntu — ```apt-get instal vowpal-wabbit```\n",
    "* Mac OS — ```port install vowpal_wabbit```\n",
    "* Windows — скачать установочник [тут](https://github.com/eisber/vowpal_wabbit/releases)\n",
    "\n",
    "[Варианты](https://github.com/JohnLangford/vowpal_wabbit/wiki/Download) установки из официальной wiki"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Проверяем VW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Num weight bits = 18\r\n",
      "learning rate = 0.5\r\n",
      "initial_t = 0\r\n",
      "power_t = 0.5\r\n",
      "using no cache\r\n",
      "Reading datafile = \r\n",
      "num sources = 1\r\n",
      "\r\n",
      "\r\n",
      "VW options:\r\n",
      "  --random_seed arg                     seed random number generator\r\n",
      "  --ring_size arg                       size of example ring\r\n",
      "\r\n",
      "Update options:\r\n",
      "  -l [ --learning_rate ] arg            Set learning rate\r\n",
      "  --power_t arg                         t power value\r\n",
      "  --decay_learning_rate arg             Set Decay factor for learning_rate \r\n"
     ]
    }
   ],
   "source": [
    "!vw -h | head -n10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Формат ввода\n",
    "\n",
    "* Использует специальный формат входных данных\n",
    "* ```Label [weight] |Namespace Feature ... |Namespace ...```\n",
    "    * ```Label``` — метка класса для задачи классификации или действительное число для задачи регрессии\n",
    "    * ```weight``` — вес объекта, по умолчанию у всех 1\n",
    "    * ```Namespace``` — все признаки разбиты на области видимости, может использоваться для раздельного использования или создания квадратичных признаков между областями\n",
    "    * ```Feature``` — ```string[:value]``` или ```int[:value]``` строки будут хешированы, числа будут использоваться как индекс в векторе признаков. ```value``` по умолчанию равно $1$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Hashing trick\n",
    "\n",
    "Вводится функция $h$, с помощью которой получается индекс для записи значения в вектор признаков объекта.\n",
    "\n",
    "$$h : F \\rightarrow \\{0, \\dots, 2^b - 1\\}$$\n",
    "\n",
    "С помощью ```--b``` можно задавать размер области значений хеш-функции. Чем больше значение ```b```, тем меньше вероятность получить коллизии при хешировании признаков."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Optimization\n",
    "\n",
    "Может использовать ```SGD``` или ```L-BFGS``` (квази-ньютоновский метод второго порядка, подробнее про работу [оптимизации](http://aria42.com/blog/2014/12/understanding-lbfgs))\n",
    "\n",
    "* ```SGD``` по умолчанию. Почти всегда необходимо несколько проходов по данным.\n",
    "* ```L-BFGS``` включается с помощью ```--bfgs```, работает гораздо медленнее и только для выборок небольшого размера\n",
    "* Количество проходов для ```SGD``` задаётся с помощью параметра ```--passes```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Параметры оптимизации\n",
    "\n",
    "Проходим по всем элементам обучающей выборки много раз, на каждом объекте обновляем веса:\n",
    "<br/>\n",
    "\n",
    "\n",
    "$$w_{t+1} = w_{t} + \\eta_t \\nabla_{w}\\ell(w_{t}, x_{t})$$\n",
    "$$\\eta_t = \\lambda d^k \\left( \\frac{t_0}{t_0 + t} \\right)^p$$\n",
    "\n",
    "Здесь $t$ — номер объекта при обучении, $k$ — номер эпохи.\n",
    "\n",
    "* $\\lambda$: ```-l```\n",
    "* $d$: ```--decay_learning_rate```\n",
    "* $t_0$: ```--initial_t```\n",
    "* $p$: ```--power_t```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Оценка качества\n",
    "\n",
    "Перед учётом очередного элемента в обучении на нём считается ошибка $e_i$. Далее эта ошибка усредняется со всеми предыдущими, получая текущую оценку качества.\n",
    "\n",
    "$$\\text{progressive error} = \\frac{e_1 + e_2 + ... + e_s}{s}$$\n",
    "\n",
    "$e_i$ — loss на объекте $x_i$ при обучении на объектах $\\{x_1 ... x_{i-1}\\}$\n",
    "\n",
    "\n",
    "Подробнее про использование [progressive validation](http://hunch.net/~jl/projects/prediction_bounds/progressive_validation/coltfinal.pdf)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Функции потерь\n",
    "\n",
    "Задаются через опцию ```--loss_function```\n",
    "\n",
    "<img src=\"files/images/loss.png\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Регуляризация \n",
    "\n",
    "Задаётся через два флага\n",
    "* ```--l1```\n",
    "* ```--l2```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Квадратичные признаки\n",
    "\n",
    "* ```-q ab``` — создаёт квадратичные признаки, перемножая все признаки из областей видимости, названия которых начинаются на букву __a__ и на букву __b__\n",
    "* ```--ignore a``` — игнорирует все признаки из области видимости, название которой начинается на букву __a__\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Режим демона\n",
    "\n",
    "* ```--daemon``` — запускает __vw__ в режиме сервиса на порту, который можно задать с помощью ```--port```\n",
    "* Позволяет обучать модель и/или применять модель по сети"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Примеры для бинарной классификации\n",
    "\n",
    "* Два класса с признаками A и B\n",
    "\n",
    "<pre>\n",
    "-1 | A:1 B:10\n",
    " 1 | A:-1 B:12\n",
    "</pre>\n",
    "\n",
    "* Можно использовать текст без обработки\n",
    "\n",
    "<pre>\n",
    "-1 | so i find myself porting a game that was originally written\n",
    " 1 | i ve been using tortoisesvn in a windows environment for quite some time\n",
    "</pre>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## ```windows vs linux```\n",
    "\n",
    "Оставим только вопросы, в которых есть тег __windows__ или __linux__ и сконвертируем данные в формат подходящий для ```vw```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-1 | as the title says does any unix like system a\r\n",
      "-1 | i want to link ln s all files that are in mnt\r\n",
      "1 | in windows xp 2003 there is one entry in the r\r\n",
      "-1 | how could i forward the x window output from \r\n",
      "-1 | as the title says what s your ide of choice t\r\n"
     ]
    }
   ],
   "source": [
    "texts = pd.read_csv('windows_vs_linux.10k.tsv', sep='\\t', header=None)\n",
    "texts[1].replace({0: '-1 ', 1: '1 '}, inplace=True)\n",
    "\n",
    "train_texts, test_texts = train_test_split(texts)\n",
    "\n",
    "\n",
    "train_texts[[1, 0]].to_csv('win_vs_lin.train.vw', sep='|', header=None, index=False)\n",
    "test_texts[[1, 0]].to_csv('win_vs_lin.test.vw', sep='|', header=None, index=False)\n",
    "\n",
    "!head -n 5 win_vs_lin.train.vw | cut -c 1-50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Обучение"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "!vw -d win_vs_lin.train.vw -f model.vw --loss_function logistic  --passes 100 -c -k 2> train.log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ```-d``` — откуда брать данны для обучения\n",
    "* ```-f``` — куда сохранять модель\n",
    "* ```--passes``` — максимальное количество проходов по выборке\n",
    "* ```-c``` — создавать файл с кешем, необходимо указывать, если используется ```--passes```\n",
    "* ```-k``` — перед запуском очищать кеш"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Лог обучения"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "final_regressor = model.vw\r\n",
      "Num weight bits = 18\r\n",
      "learning rate = 0.5\r\n",
      "initial_t = 0\r\n",
      "power_t = 0.5\r\n",
      "decay_learning_rate = 1\r\n",
      "creating cache_file = win_vs_lin.train.vw.cache\r\n",
      "Reading datafile = win_vs_lin.train.vw\r\n",
      "num sources = 1\r\n",
      "average  since         example        example  current  current  current\r\n",
      "loss     last          counter         weight    label  predict features\r\n",
      "0.693147 0.693147            1            1.0  -1.0000   0.0000      101\r\n",
      "0.632252 0.571357            2            2.0  -1.0000  -0.2605       28\r\n",
      "0.652016 0.671781            4            4.0  -1.0000  -1.6879      135\r\n",
      "0.882366 1.112717            8            8.0   1.0000   0.0338       19\r\n",
      "0.848898 0.815429           16           16.0  -1.0000  -0.5651       69\r\n",
      "0.886803 0.924707           32           32.0   1.0000  -0.3310      161\r\n",
      "0.793671 0.700539           64           64.0   1.0000   1.3198      197\r\n",
      "0.721078 0.648484          128          128.0   1.0000  -0.5368       16\r\n",
      "0.634201 0.547324          256          256.0   1.0000   2.3615      208\r\n"
     ]
    }
   ],
   "source": [
    "!head -n20 train.log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Применение"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.072026\r\n",
      "2.981259\r\n",
      "2.415733\r\n"
     ]
    }
   ],
   "source": [
    "!vw -d win_vs_lin.test.vw -i model.vw -t -p output.csv --quiet\n",
    "!head -n3 output.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ```-i``` — путь до готовой модели\n",
    "* ```-t``` — не обучаться, только вернуть предсказания\n",
    "* ```-p``` — путь до файла для записи предсказаний\n",
    "* ```--quite``` — не выводить никакую информацию в консоль"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Результат\n",
    "\n",
    "Загрузим ответы и померяем $AUC$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.96704167103343197"
      ]
     },
     "execution_count": 188,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_hat = pd.read_csv('output.csv', header=None)\n",
    "roc_auc_score(test_texts[1].replace({'-1 ': 0, '1 ': 1}), y_hat[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Многоклассовая классификация\n",
    "\n",
    "Включается с помощью флага ```--oaa n```, где $n$ число классов.\n",
    "\n",
    "Формат для меток — ```label[:weight],label[:weight],...```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4 | i want to use a track bar to change a form s o\r\n",
      "7 | i have an absolutely positioned div containing\r\n",
      "1,4 | given a datetime representing a person s bir\r\n",
      "4 | given a specific datetime value how do i displ\r\n",
      "7,9 | is there any standard way for a web server t\r\n"
     ]
    }
   ],
   "source": [
    "texts = pd.read_csv('multi_tag.10k.tsv', header=None, sep='\\t')\n",
    "texts.columns = ['text', 'tags']\n",
    "classes = np.arange(21)\n",
    "texts['tags'] = map(lambda row: ','.join(map(str, 1 + classes[row.astype('bool')])) + ' ', y)\n",
    "texts[['tags', 'text']].to_csv('multi_tag.vw' , sep='|', header=None, index=False)\n",
    "\n",
    "!head -n 5 multi_tag.vw | cut -c 1-50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Обучение"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "!vw -d multi_tag.vw  -f model.vw --loss_function logistic --oaa 21 --quiet --passes 100 -c -k"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Предсказание\n",
    "\n",
    "Для простоты получим ответы на ```train```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1:-2.48879 2:-2.2027 3:-3.81814 4:1.25513 5:-3.72094 6:-6.52962 7:-5.75792 8:-4.49483 9:-3.05773 10:-6.86743 11:-6.08145 12:-1.01415 13:-6.79665 14:-4.62895 15:-4.18816 16:-3.87795 17:-5.31658 18:-4.34753 19:-2.65281 20:-6.11897 21:-4.71674\r\n",
      "1:-5.77629 2:-4.70643 3:-8.05376 4:-2.97062 5:-8.84354 6:-10.7255 7:2.6422 8:-5.72566 9:-3.39589 10:-8.78461 11:-7.98634 12:-3.78309 13:-6.47153 14:-9.07742 15:-10.1064 16:-9.9607 17:-10.6715 18:-11.5456 19:-9.8694 20:-8.54887 21:-8.8077\r\n",
      "1:-1.92299 2:-3.58227 3:-3.03307 4:-2.77937 5:-4.40152 6:-3.35077 7:-3.73742 8:-4.15759 9:-3.40149 10:-2.5327 11:-3.96967 12:-0.673776 13:-3.39927 14:-2.78582 15:-4.01405 16:-2.84286 17:-4.04175 18:-4.31216 19:-4.14539 20:-3.77989 21:-3.65737\r\n"
     ]
    }
   ],
   "source": [
    "!vw -d multi_tag.vw -i model.vw -r output.csv --loss_function logistic --quiet \n",
    "!head -n3 output.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Подробнее про VW\n",
    "\n",
    "* [Вики](https://github.com/JohnLangford/vowpal_wabbit/wiki/)\n",
    "* Раздел [Tutorial](https://github.com/JohnLangford/vowpal_wabbit/wiki/Tutorial)  на вики\n",
    "* [Pdf](https://github.com/JohnLangford/vowpal_wabbit/wiki/v7.0_tutorial.pdf) с описанием большинства возможностей VW"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}