ExampleUsage.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f2ec10a0",
   "metadata": {},
   "source": [
    "# Example usage of ease.ml/snoopy for feasibility study of ML applications"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51188570",
   "metadata": {},
   "source": [
    "This example jupyter notebook illustrates how to use the ease.ml/snoopy library to perform an automatic feasibility study."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90a55cf5",
   "metadata": {},
   "source": [
    "## Required imports\n",
    "\n",
    "Let us start with defining the required imports in order to be able to define the target dataset and the set of pre-trained models (embeddings) used to perform the feasibility study"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5e546d84",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[1;34m[18:25:12]\u001b[0m\u001b[1;33m WARNING Snoopy may use all CPU cores! Set env. variable 'SNOOPY_LIMIT_CPU' to 'y' to limit CPU usage. \u001b[0m\u001b[1;34m(__init__ -> _init_tf_pytorch on MainThread)\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "from collections import OrderedDict\n",
    "from tensorflow_datasets import Split\n",
    "\n",
    "from snoopy.embedding import EmbeddingConfig, uni_se\n",
    "from snoopy.reader import TFDSTextConfig"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae49208c",
   "metadata": {},
   "source": [
    "## Configs\n",
    "We next define the dataset (train and test splits) to fuse into the system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1affce36",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_name = \"imdb_reviews\"\n",
    "train_data_config = TFDSTextConfig(dataset_name=dataset_name, split=Split.TRAIN)\n",
    "test_data_config = TFDSTextConfig(dataset_name=dataset_name, split=Split.TEST)\n",
    "classes = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62daf18d",
   "metadata": {},
   "source": [
    "Then, we define a set of pre-trained models (embeddings) we want to use. Naturally, as we are taking the min over all possible transformations, the more models we specify here the better. Note nevertheless, that each additional model will require more compute time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ac3e8ae5",
   "metadata": {},
   "outputs": [],
   "source": [
    "models = OrderedDict({\n",
    "    \"use\": EmbeddingConfig(uni_se, batch_size=10, prefetch_size=1)\n",
    "}) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b38faa7f",
   "metadata": {},
   "source": [
    "Finally, define a target error (1 - target accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3509960e",
   "metadata": {},
   "outputs": [],
   "source": [
    "target = 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6057619",
   "metadata": {},
   "source": [
    "## Run Snoopy\n",
    "\n",
    "### Addtional imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "42721854",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import os\n",
    "import os.path as path\n",
    "import torch as pt\n",
    "\n",
    "from snoopy import set_cache_dir\n",
    "from snoopy.pipeline import run, store_embeddings\n",
    "from snoopy.result import BerStoringObserver\n",
    "from snoopy.strategy import SimpleStrategyConfig\n",
    "from snoopy.reader import ReaderConfig, data_factory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38a729f9",
   "metadata": {},
   "source": [
    "### Snoopy settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7d9be767",
   "metadata": {},
   "outputs": [],
   "source": [
    "set_cache_dir(\"cache\")\n",
    "\n",
    "results_folder = \"results\"\n",
    "if not path.exists(results_folder):\n",
    "    os.mkdir(results_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "315743d4",
   "metadata": {},
   "source": [
    "### Run with Bayer Error Rate (BER) Storing Observer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "91384edd",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:No config specified, defaulting to first: imdb_reviews/plain_text\n",
      "INFO:absl:Load dataset info from cache/TFDS_cache/imdb_reviews/plain_text/1.0.0\n",
      "INFO:absl:Reusing dataset imdb_reviews (cache/TFDS_cache/imdb_reviews/plain_text/1.0.0)\n",
      "INFO:absl:Constructing tf.data.Dataset imdb_reviews for split test, from cache/TFDS_cache/imdb_reviews/plain_text/1.0.0\n",
      "\u001b[1;34m[18:25:12]\u001b[0m\u001b[2;37m DEBUG   Loaded text dataset imdb_reviews (split: test) with 25000 points \u001b[0m\u001b[1;34m(tfds -> read_data on MainThread)\u001b[0m\n",
      "INFO:absl:No config specified, defaulting to first: imdb_reviews/plain_text\n",
      "INFO:absl:Load dataset info from cache/TFDS_cache/imdb_reviews/plain_text/1.0.0\n",
      "INFO:absl:Reusing dataset imdb_reviews (cache/TFDS_cache/imdb_reviews/plain_text/1.0.0)\n",
      "INFO:absl:Constructing tf.data.Dataset imdb_reviews for split train, from cache/TFDS_cache/imdb_reviews/plain_text/1.0.0\n",
      "\u001b[1;34m[18:25:12]\u001b[0m\u001b[2;37m DEBUG   Loaded text dataset imdb_reviews (split: train) with 25000 points \u001b[0m\u001b[1;34m(tfds -> read_data on MainThread)\u001b[0m\n",
      "INFO:absl:No config specified, defaulting to first: imdb_reviews/plain_text\n",
      "INFO:absl:Load dataset info from cache/TFDS_cache/imdb_reviews/plain_text/1.0.0\n",
      "INFO:absl:Reusing dataset imdb_reviews (cache/TFDS_cache/imdb_reviews/plain_text/1.0.0)\n",
      "INFO:absl:Constructing tf.data.Dataset imdb_reviews for split test, from cache/TFDS_cache/imdb_reviews/plain_text/1.0.0\n",
      "\u001b[1;34m[18:25:12]\u001b[0m\u001b[2;37m DEBUG   Loaded text dataset imdb_reviews (split: test) with 25000 points \u001b[0m\u001b[1;34m(tfds -> read_data on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:12]\u001b[0m\u001b[1;32m INFO    Initializing https://tfhub.dev/google/universal-sentence-encoder/4 on CPU:0 \u001b[0m\u001b[1;34m(tf_hub -> get_instance on MainThread)\u001b[0m\n",
      "INFO:absl:Using cache/TensorFlow_cache to cache modules.\n",
      "\u001b[1;34m[18:25:16]\u001b[0m\u001b[2;37m DEBUG   Preparing test dataset for embedding 'use' \u001b[0m\u001b[1;34m(simple -> execute on MainThread)\u001b[0m\n",
      "/Users/rengglic/repos/snoopy-public/snoopy/embedding/base.py:185: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /Users/distiller/project/conda/conda-bld/pytorch_1639180724445/work/torch/csrc/utils/tensor_numpy.cpp:189.)\n",
      "  current_labels = pt.as_tensor(current_labels_np, dtype=pt.int64, device=self._device).view((-1, 1))\n",
      "\u001b[1;34m[18:25:18]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 100 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:19]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 200 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:20]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 300 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:21]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 400 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:22]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 500 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:23]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 600 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:24]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 700 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:25]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 800 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:26]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 900 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:27]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 1000 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:28]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 1100 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:29]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 1200 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:30]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 1300 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:31]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 1400 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:32]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 1500 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:33]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 1600 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:34]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 1700 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:35]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 1800 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:36]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 1900 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:37]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 2000 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:38]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 2100 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:39]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 2200 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:40]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 2300 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:41]\u001b[0m\u001b[2;37m DEBUG   Batches processed: 2400 \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:42]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.001 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:42]\u001b[0m\u001b[2;37m DEBUG   Test dataset for embedding 'use' is now prepared \u001b[0m\u001b[1;34m(simple -> execute on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:42]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=0, err=25000) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:43]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.005 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:44]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=1000, err=7549) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:45]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:45]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=2000, err=7543) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:46]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:47]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=3000, err=7375) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:48]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:48]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=4000, err=7358) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:49]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:49]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=5000, err=7312) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:50]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:51]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=6000, err=7189) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:51]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:52]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=7000, err=7151) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:53]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:53]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=8000, err=7045) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:54]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:55]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=9000, err=6956) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[1;34m[18:25:56]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:56]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=10000, err=6964) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:57]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:57]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=11000, err=6876) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:58]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:25:59]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=12000, err=6834) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:00]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:00]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=13000, err=6790) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:01]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:01]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=14000, err=6866) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:02]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:03]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=15000, err=6832) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:04]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:04]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=16000, err=6832) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:05]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:06]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=17000, err=6816) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:07]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:07]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=18000, err=6810) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:08]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:09]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=19000, err=6806) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:10]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:10]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=20000, err=6806) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:11]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:11]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=21000, err=6792) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:12]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:13]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=22000, err=6770) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:14]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:14]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=23000, err=6749) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:15]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:16]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=24000, err=6724) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:17]\u001b[0m\u001b[2;37m DEBUG   Average feature retrieve time was  0.000 seconds \u001b[0m\u001b[1;34m(base -> _compute_embedding on MainThread)\u001b[0m\n",
      "\u001b[1;34m[18:26:17]\u001b[0m\u001b[1;32m INFO    Current result: \n",
      "\tuse: PR(n=25000, err=6711) \u001b[0m\u001b[1;34m(result -> on_update on MainThread)\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "test_data = data_factory(test_data_config)\n",
    "test_size = float(test_data.size)\n",
    "\n",
    "observer = BerStoringObserver(classes, test_size)\n",
    "\n",
    "run(train_data_config=train_data_config,\n",
    "    test_data_config=test_data_config,\n",
    "    embedding_configs=models,\n",
    "    strategy_config=SimpleStrategyConfig(train_size=1000, test_size=5_000),\n",
    "    observer=observer,\n",
    "    device=pt.device(\"cpu\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53bf788e",
   "metadata": {},
   "source": [
    "### Store the results into the results folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "90e375a3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[1;34m[18:26:17]\u001b[0m\u001b[2;37m DEBUG   Storing embedding 'use' to file 'use' \u001b[0m\u001b[1;34m(result -> store on MainThread)\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "observer.store(results_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35cee9a6",
   "metadata": {},
   "source": [
    "## Use the output of snoopy (BER estimate and plots)\n",
    "\n",
    "As snoopy does output results in files, we next show how to use the results (BER estimate) and plots to guid users towards feasibility study."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2197dd02",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minimal error achievable is 0.1597\n"
     ]
    }
   ],
   "source": [
    "# Set the random baseline as min achievable error first, then iterate over all models\n",
    "min_error = (classes - 1.0) / float(classes)\n",
    "for k in models.keys():\n",
    "    f = path.join(results_folder, \"{0}.npz\".format(k))\n",
    "    if path.exists(f):\n",
    "        items = np.load(f)\n",
    "        error = items['ber'][-1]\n",
    "        min_error = min(min_error, error)\n",
    "\n",
    "print('Minimal error achievable is {0:.4f}'.format(min_error))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dccb8920",
   "metadata": {},
   "source": [
    "This means, that every target error above this value would be treated as _feasible_, and every target error below this value as _infeasible_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7324251c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Target Achievable:  False\n"
     ]
    }
   ],
   "source": [
    "print('Target Achievable: ', target > min_error)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50106f90",
   "metadata": {},
   "source": [
    "To trust, or ignore the signal of snoopy, one typically also inspects the convergence plots:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "cb57f5fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[None, None]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdkAAAFMCAYAAACH7MbyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABVD0lEQVR4nO3deXxM5/7A8c/MJJN9F2silhAqYqdBSC2N0EW0iqql1l+rm+W2klttKW2paylui1qqekvdlpYKqiluqRZtVRWlQmIPkcg+k5nz+2NkakwmEjImie/79fJinuc5Z57zdfjmnPOc51EpiqIghBBCiHKndnQHhBBCiKpKkqwQQghhJ5JkhRBCCDuRJCuEEELYiSRZIYQQwk4kyQohhBB2IklWCCGEsBMnR3egort6NQej8fZfJQ4I8OTKlexi6y6m5/Kvtb/y1INhRDQMuO3vqKxKis29TmJjm8TGNomNbeURG7VahZ+fR5m2kSR7C0ajckdJtmgfxVGrVFy6mkf6tfw7/o7K6l497tKQ2NgmsbFNYmObI2Ijt4sdyM1FA0C+zuDgngghhLAHSbIO5KK9nmQLCh3cEyGEEPYgt4sdSKNWE9elAY2DfBzdFSGEEHYgSdbBHu5Yz9FdEEIIYSdyu9jBMnN0XM0qcHQ3hBBC2IFcyTrYvM8O4uOp5aX+LRzdFSFECfR6HVlZGRQW6jAaK+ZgxUuX1BiNRkd3o0K6VWw0Gic8PX1xcyvbKzq3IknWwdxcNDLwSYgKLi8vh6ysq3h6+uDi4o9arUGlUjm6W1acnNQUFkqSLU5JsVEUBb1eR0ZGGkC5Jlq5Xexgrlon8uQVHiEqtOzsTHx9q+Hu7oVG41QhE6y4fSqVCq3WBV/fQLKzM8p13w5Psps2baJPnz5EREQQGxvLhg0bSmx/6dIlJk2aRGRkJK1bt+bZZ5/l9OnTFm0KCwuZN28eXbt2pUWLFjz55JP89ttvdjyK2+fqoiFfJ1eyQlRkBoMeZ2cXR3dD2JmzsxaDoXz/P3Zokk1MTGTSpEl06tSJRYsW0b59e1555RW2bNlSbPuCggJGjRrFoUOHeO211/jXv/7FpUuXeOqpp7h27Zq53YwZM1i5ciWjR49m7ty5aDQahg8fTmpq6t06tFJz0zqRVyBXskJUdHL1WvXZ4+/Yoc9k58yZQ2xsLAkJCQBERUWRmZnJ/Pnz6dWrl1X77777jmPHjvH5558THh4OQKNGjejevTtbt26lf//+nDlzhrVr1zJlyhQGDRoEQOfOnYmJieHDDz9k6tSpd+8AS6F90+qE1PRydDeEEELYgcOuZFNTU0lJSeHBBx+0KI+JieHkyZPFXnV27tyZ//znP+YEC+Ds7AyATqcDYO/evRgMBmJiYsxttFot0dHR7Nq1yx6HckfC6vrRpUVtR3dDCCGEHTgsyZ48eRKA+vXrW5SHhIQAkJycbLWNp6cnbdq0AUCv13P06FEmT56Mr68vPXv2NO/Xx8cHf39/q/2eO3eO/Pz8cj+WO5GTr+f0hSwKDTIiUAghqhqH3S7OysoCTInzRh4epqHT2dklL0n0/PPP891336FWq5kxYwbVq1c3b3fzPm/cb05ODq6urqXuZ0CA9b7KKjDQ9u3ggz+lMH/tLyxN6EFgQPm+n1UZlBSbe53Exra7HZtLl9Q4OTl8nGipVJZ+OkJpYqNWq8v1/HJYklUU05JDNz9oLipXq0sOxujRoxk2bBhfffUV8fHxAPTr18+8fWm/71auXMm+o+WRAgO9SEvLsllfqNMDcPZ8Jpp77CXyW8XmXiaxsc0RsTEajZXi/VN5T9a20sbGaDTaPL/UalWZL7wclmS9vEw/Kdx8xZqTk2NRb0vRbePIyEjOnj3L4sWL6devH56enuZ9FLff4q5yHcnVxfRXIMvdCSFE1eOwJFv0LDYlJYWwsDBzedE7rzc/qwX4448/SE5Opk+fPhblzZo149ChQwA0aNCAjIwMMjMz8fH5e3Wb06dPExQUhFarLfdjuROuRcvdybuyQlRKMz/52aqsXdPqdGsdRIHewLzPDlrVd2pei84RtcjK1fHv9b9b1T/Qug7tm9Yg/Vo+Szf+YVUf074uLRtV4/yVHFZtOcYrg1uXz8GIcuewm/chISEEBQVZvRO7bds26tWrR+3a1iNu9+7dy8SJE0lJSTGXGQwG9u7dS+PGjQHo2LEjAFu3bjW30el07Ny501xXkbhpTT/nyLuyQghR9agUWw8x74IvvviC+Ph4Bg8eTHR0NElJSXz66afMnTuX3r17k56eTkpKCqGhoXh6epKRkUFcXBzu7u48//zzuLq68sknn7Bnzx6WL19Ohw4dAJg8eTKbN29mwoQJhISEsGLFCn7//XfWr19vHr1cWvZ+JptXUMjPf6YRFuxLNV+32/6eykieO9omsbHNEbG5cOE0NWuW7f8ORyjrM9mff97P3Lmz+Pjjzyw+T58+i3femUZBgQ5QeOihvvTr1x+Ajz5axs6dSRiNCrVq1WLixMlUqxZoj8MpV6WNTUl/15XqmSyYBirpdDqWL1/OunXrCA4OZubMmfTu3RuAHTt2EB8fz6pVq+jQoQO+vr6sXr2a2bNnM23aNHJycoiIiOCjjz6ibdu25v1OmzYNb29vlixZQm5uLs2aNWPFihVlTrB3g5uLE52a13J0N4QQwuw//1lFx45dGDJkOFeuXOa99/5F376PsXXrZk6ePMGSJR/h5OTEl19+wTvvvMns2e85ussVlsNX4Rk4cCADBw4stq5fv37069fPoqxOnTrMnTu3xH1qtVoSEhLMM0lVdCfOZOLl7kwNf3dHd0UIIejS5QGmT3+dI0cO07Zte1566R+o1Wr27PmeI0cOM2rUUACMRkOFm3ugonF4khXwr89+pWuL2gzs3sjRXRFC3ENUKhU3PjAsLDQNwOzUKYo1a75g374fOXBgHytWLGXZso8xGg0MHjyMuLjHAdN4l6ysa8XtWlwnby1XAK5aDXmypqwQ4i7z9fXj4sULXL2ajqIobN9uGjD6xhv/5Ntvv6FHjxgmTpyMh4cHZ8+eoX37SDZu3EBOjunVyw8//IA333zNkYdQ4cmVbAXgpnWS92SFEHdd/foNePTRfowcOYSAgGp06hTFkSOHGT58FDNnvsmXX36BRqOmS5doWrZsTYsWrbh8OY2xY58GVNSoUZN//vMNRx9GhebQ0cWVgb1HFwNMW7kPT3dnJjzR8ra/pzKSEbS2SWxsk9HFtsmMT7Y5anSx3C6uANxc5EpWCCGqIrldXAHEdWng6C4IIYSwA0myFUBoHZ9bNxJCCFHpyO3iCuDc5Rx+OZ7m6G4IIYQoZ5JkK4AfDl/g3+t/t7lMnxBCiMpJkmwF4KrVYDAqFBpkVKAQQlQlkmQrAFdZiUcIIaokSbIVgKwpK4QQVZMk2QrAzUWuZIUQoiqSJFsBNAryYfLg1tTwv7fWkxVCiKpO3pOtALzctXi5ax3dDSGEEOVMrmQrgHxdIXt+P8+F9FxHd0UIIUQ5kivZCiCvwMCHm44wNCaMmrJwuxDiLpg3711+/fUXAE6dOkmtWrVxcXEFYPHi5eY/l7fs7GwSEibx3nsfWNVt3ryR+fNnU6tWHYvyhg1DmTJlml36Y2+SZCsANxfT6OI8GV0sRKXTt6/1WIpHHilkxAg9ubnw5JPW9QMH6hk4sJArV1SMHGmdzIYP19O3byFnz6oYN866/plndMTEGDhxQsWkSa5s2JBX5n6/9NI/zH9+/PGHef316TRpcl+Z91NWWVnXOHLksM36Fi1aMWvWPLv3426RJFsBuDhrUAH5MrpYCOFgeXl5zJ79NmfOpJKZmYm7uztvvDGdunXr8dxzY/D29iEl5RR9+z7G/fd34u23p3HtWiYBAdVQFIiJiaV374c5dOgg77+/gPz8PNRqDU8/PZpOnaJ4662pFBQUMHz4kyxb9jEajabUfZsx4w2uXcvk7NmzdOzYmatX0y0+Dx06gjlzZnL8+DFUKhX339+RMWPG4eTkRFRUBzp37sqJE3/etR8oQJJshaBSqXB10ciVrBCVUElXke7uJdcHBCgl1tepU3J9aGjJ9bdj797deHl5sXjxCgDeffctPv/8M8aPfxkALy8vVq9eB8DYsU/Tq1cf4uIe59SpZEaNGkJMTCzXrl3jrbemMmfOQmrVqs3ly2mMGTOchg0bkZDwOkOHDmDlyv8U+/0HD/7C8OFPWpT17z+QPn0eASA/v4DVqz8DTEn3xs/Tp7+Ot7cPq1atRa/XM3nyBD79dDVDhgxHr9fTqVMUb775TrnG61YkyVYQrlonuZIVQjjcAw/0oHbtIP773zWcOXOGX345QHh4hLm+RYtWAFy7Zrrtu2jRUgDq1atPmzbtADh8+DeuXLlCfPwki33/9ddxGjRoWOL33+p2cUREC5uf9+7dw/vvL0OlUqHVann00cdYt+5ThgwZbtH3u0mSbAUxYUBL3F3kr0MI4Vjr1/+Xr776gscee4KePXvh7e3N+fPnzPVubqZnzE5Optu8Ny5solabygwGIyEh9Vi69CNz3eXLafj6+pGWdumO+ufm5m7zs6IYUalUFp8LCwttbns3yCs8FUSdah74ebk4uhtCiHvcTz/9QGzswzz0UF/q1g1h9+7/YTRa32Vzd/egefMWfP31VwCcO3eWAwf2oVKpaNasOWfOpPLrrz8DcPz4MQYOjCMt7RIajQaDwWiXVcfat4/k888/Q1EUdDodX321nnbtOpT795SFXDpVEAdPXCa3oJDIZjUd3RUhxD1s4MAhvPvuDL7++ksURaFZswhOnjxRbNtXX53KO++8yfr1/yUwMND8GpCfnx8zZsxi0aL56HQ6FMXIlCnTqFWrNgaDgfvua8aQIU+waNFSfHx8LfZZ3DNZjUbDsmUf37LvL700iblz32Xo0AHo9YXcf38kQ4eOuO1YlAeVIouYlujKlWyMxtsPUWCgF2lpWbdst+iLQ1xIz+XNUY79qetuKm1s7kUSG9scEZsLF05Ts2bIXf3O2+HkpKaw8O4tmfnRR8uIju5OSEg9srOzGTZsILNnv0f9+g3uWh9Kq7SxKenvWq1WERDgWbbvLVNrYTeuLhpZhUcIUakEB4fw2mvxqNUqDAYDTz01vEImWEeSJFtBuGqdZBUeIUSl0q1bD7p16+HoblRoMvCpgnBz0ZCvM9hlMIAQQgjHkCRbQbhqnTAqCrq7+DxFCCGEfTn8dvGmTZt4//33SU1NpU6dOowdO5a+ffvabJ+Wlsb8+fPZvXs3GRkZ1K9fn9GjRxMbG2tus3//fgYPHmy1bXR0NIsXL7bHYdyx6Ja1iWxWE2cn+blHiIpIURSLdzBF1WOPO4kOTbKJiYlMmjSJoUOHEhUVxfbt23nllVdwdXWlV69eVu11Oh2jRo0iKyuLF154gerVq7N161ZeeuklDAYDDz30EADHjh3D3d2dFStWWGzv7e19V47rdri7OuNun0UvhBB3SKNxRq8vQKuVf6RVmV6vQ6Mp37To0CQ7Z84cYmNjSUhIACAqKorMzEzmz59fbJLdtWsXR48eZd26dUREmKb56tSpE+fOnWPp0qXmJHv06FEaNWpEy5Yt79qx3KlLGXn88PsFoiJq4e8t/5CFqEg8PX3IyLiMh4cPrq5uqNUauaqtQhRFQa/XkZGRhpeXX7nu22FJNjU1lZSUFCZMmGBRHhMTQ2JiIqmpqQQHB1vUeXh4MGDAAJo3b25R3qBBAw4cOGD+fOTIEZo2bWq/ztvBlcx8vvw+mcbBvpJkhahg3Nw8cHJyJjs7g5yczGJnQKoI1Go1RqOM6yjOrWKj0Tjh5eWHm5tHuX6vw5LsyZMnAahfv75FeUiI6SXg5ORkqyQbGRlJZGSkRZler2fnzp00atQIAKPRyPHjx/Hz8yMuLo7jx49TrVo1hg4dytNPP11hf/osWlNW3pUVomJydtbi51fd0d0okUxiYpujYuOwJJuVZTpYT0/L2TM8PEw/RWRnZ5dqP7Nnz+bUqVMsWrQIMCXn/Px8kpOTmTBhAn5+fnz77bfMmjWL7OxsXnjhhXI8ivLjpjX9VchKPEIIUXU4LMkWjeK6+cqyqFytLnmUraIovPvuu6xcuZKRI0fSo4fphegaNWqwdOlSmjZtSmBgIGC6As7Pz2fp0qWMGDHCKrGXpKxTaBUnMNDrlm2cXJ1Nv7s4lap9VXEvHWtZSWxsk9jYJrGxzRGxcViS9fIyHezNV6w5OTkW9cXR6XRMnjyZr7/+mpEjR/Lyyy+b6zw9PenSpYvVNtHR0axbt47k5GSrZ7oluVtzFxfoTVewaVdy7pnbPXJryzaJjW0SG9skNraVR2wq1dzFRc9iU1JSCAsLM5efPn3aov5m2dnZjB07lp9//pmEhASGDRtmUX/s2DEOHDhA//79cXZ2Npfn5+cD4OdXviPHyovWSc3Cl6Jw1Tr81WUhhBDlxGEzH4SEhBAUFMSWLVssyrdt20a9evWoXbu21TYGg4FnnnmGgwcPMmfOHKsEC6YkPXXqVHbt2mVRvnnzZoKCgqhTp075Hkg5UalUuLs6o1ZXzIFZQgghys6hl03jxo0jPj4eHx8foqOjSUpKIjExkblz5wKQnp5OSkoKoaGheHp6smbNGn766ScGDBhArVq1+PXXX837UqlUtGjRgujoaMLDw5kyZQrp6enUrFmTjRs3kpSUxIIFCyrs6GKAzXtP4+OhpVPzWo7uihBCiHLg0CTbr18/dDody5cvZ926dQQHBzNz5kx69+4NwI4dO4iPj2fVqlV06NCBrVu3ArB27VrWrl1rsS+NRsMff/yBVqtl6dKlzJs3j4ULF5Kenk6jRo1YuHCheXBURfXD7xeo6e8uSVYIIaoIWbT9Fu7WwCeAGR/vx8VZw6SBrW77+yoTGaRhm8TGNomNbRIb2xw18Elmo69A3GRNWSGEqFIkyVYgrlqNzPgkhBBViCTZCsTVxQm9rCcrhBBVhryUWYEM79VEXuERQogqRK5kKxBJsEIIUbVIkq1A/jiVzrJNf5inWBRCCFG5SZKtQC5dzWP37xfIzZfBT0IIURVIkq1AXLWypqwQQlQlZR74lJKSQlpams0V5tu1a3fHnbpXubpcX1NWJ7eLhRCiKih1kj179izjx4/n0KFDxdYrioJKpeLIkSPl1rl7jdv1K9m8ArmSFUKIqqDUSfbtt9/m8OHDDBgwgKZNm6LVau3Zr3uSm4sT7i5OGO5gGkchhBAVR6mT7J49exg2bJjFAumifNWt4cXC8dYLzgshhKicSj3wycnJibp169qzL0IIIUSVUuokGxUVRVJSkj37cs8zGI38e8Pv/HTkoqO7IoQQohyU+nbx6NGjefbZZ3nxxRfp1asX/v7+qNXWOVpGF98+tUrFL3+mUcPPjfZNazi6O0IIIe5QqZNs3759ATh37hzbtm2zqpfRxXdOpVLhqtXI6GIhhKgiSp1k33rrLVQqmVvX3txcZE1ZIYSoKkqdZPv162fPfojrZE1ZIYSoOso045PBYGDdunVs376dc+fO4ezsTK1atXjggQd4/PHH0Wg09urnPSPQ1w13F1mBUAghqoJS/2+en5/PyJEjOXDgAJ6engQHB2M0Gtm3bx87d+5k/fr1rFq1SiapuEPPPxbh6C4IIYQoJ6VOsosWLeLAgQP84x//YOjQoTg7OwOg1+v55JNPmDlzJh988AEvvPCC3TorhBBCVCalfk/266+/Ji4ujpEjR5oTLICzszPDhw8nLi6OTZs22aWT95Lt+1NZ+EXx80MLIYSoXEqdZC9dukSLFi1s1jdv3pwLFy6US6fuZZcz8zl8Kt3R3RBCCFEOSp1kq1evzuHDh23W//7771SrVq1cOnUvc9VqKNAZMCqySIAQQlR2pU6yffr04fPPP+fjjz/GYPj7PU6DwcCqVatYv349vXr1sksn7yVu10cWF8iaskIIUemVeuDTuHHj2LdvHzNmzOC9994jODgYgNTUVLKysmjevDnPPfec3Tp6r3C9YU1ZN3mVRwghKrVS/y/u6urKxx9/zH//+1++++47zpw5g6IotGjRggceeID+/fvL6zvlwM/Lhbo1POV2sRBCVAFlulRydnZm0KBBDBo0yF79uedFNKxGREN5ti2EEFWBzSS7b98+GjZsiL+/v/lzacgqPEIIIYSJzSQ7ZMgQ3n33XR5++GHz55IWCJBVeMrHpYw8PtjwO/26NiC8foCjuyOEEOIO2Eyyb7/9Ni1btjR/ttcqPJs2beL9998nNTWVOnXqMHbsWPOyesVJS0tj/vz57N69m4yMDOrXr8/o0aOJjY01tyksLGThwoWsX7+ejIwMmjVrxuTJk4mIqPhTFqqAUxeyyMjSOborQggh7pDNJBsXF2fx+Var8BgMBs6dO1emL09MTGTSpEkMHTqUqKgotm/fziuvvIKrq2uxrwPpdDpGjRpFVlYWL7zwAtWrV2fr1q289NJLGAwGHnroIQBmzJjB+vXrmTRpErVr12bFihUMHz6cL7/80jwquqIqGlEsK/EIIUTlV+r3ZJs2bVritInr168v8Qq0OHPmzCE2NpaEhASioqKYOnUqsbGxzJ8/v9j2u3bt4ujRo8ybN4+4uDg6derEtGnTiIqKYunSpQCcOXOGtWvX8sorr/DUU0/RrVs3li1bho+PDx9++GGZ+ucI5ld45D1ZIYSo9GxeyV68eJEffvjB/FlRFPbt20dhofUVltFoZOPGjWW6nZyamkpKSgoTJkywKI+JiSExMZHU1FSrq04PDw8GDBhA8+bNLcobNGjAgQMHANi7dy8Gg4GYmBhzvVarJTo6mh07dpS6f47ipFHjpFHLlawQQlQBNpOsv78/H3zwAadOnQJApVKxdu1a1q5da3NnQ4YMKfUXnzx5EoD69etblIeEhACQnJxslWQjIyOJjIy0KNPr9ezcuZNGjRqZ9+vj42MeFX3jfs+dO0d+fj6urq6l7qcj3FfPD3+vit1HIYQQt2YzyTo7O7N8+XLzpBPDhg1j7NixdOrUyaqtWq3G39+fBg0alPqLs7KyAPD09LQo9/DwACA7O7tU+5k9ezanTp1i0aJF5u1u3ueN+83JySlTkg0IsN5XWQUGepWp/YxnO9/xd1YWZY3NvURiY5vExjaJjW2OiE2Jk1HUrl2b2rVrA6bRxm3bti23gUPK9RmNbr7FXFSuVpf8uFhRFN59911WrlzJyJEj6dGjh8X2pf2+W7lyJRuj8fZnXwoM9CItLeu2t6/KJDa2SWxsk9jYJrGxrTxio1arynzhVeqBT3FxcTYTrNFoJCsri2+++abUX+zlZfqJ4uYr1pycHIv64uh0OiZOnMiyZcsYOXIkL7/8srnO09PTvI/i9lvcVW5Fs3TjH7y/4XdHd0MIIcQdKvW0itnZ2bz11lts27aN3Nxcm1eMpZ2MouhZbEpKCmFhYeby06dPW9QX14+xY8fy888/k5CQwLBhwyzqGzRoQEZGBpmZmfj4+FjsNygoqFLMr5yTryczW96TFUKIyq7UV7KzZs3iiy++IDQ0lMjISBRF4aGHHiIyMhInJydcXFxYsGBBqb84JCSEoKAgtmzZYlG+bds26tWrZ75NfSODwcAzzzzDwYMHmTNnjlWCBejYsSMAW7duNZfpdDp27txprqvoXLUaGV0shBBVQKmvZHfs2EHPnj1ZsGABV69eJTIykiFDhhAREcGRI0cYPHiwecRwaY0bN474+Hh8fHyIjo4mKSmJxMRE5s6dC0B6ejopKSmEhobi6enJmjVr+OmnnxgwYAC1atXi119/Ne9LpVLRokUL6tSpQ1xcHNOnTyc3N5eQkBBWrFhBZmYmo0aNKlP/HMXNxUnekxVCiCqg1Ek2PT3dPLLYz8+PGjVq8NtvvxEREUHTpk15/PHH2bhxI2PGjCn1l/fr1w+dTsfy5ctZt24dwcHBzJw5k969ewOmxB4fH8+qVavo0KGD+eq0uFeJNBoNf/zxBwDTpk3D29ubJUuWkJubS7NmzVixYoX59aCKzk3rJFeyQghRBZQ6yXp4eGA0Gs2f69aty59//mn+3LhxYz7//PMyd2DgwIEMHDiw2Lp+/fpZTOe4atWqUu1Tq9WSkJBAQkJCmftTEQTX8KR5gwDzogtCCCEqp1I/k42IiCAxMRGDwXQbMzQ0lP3795sHQCUnJ1eKQUWVQWSzmoyLay4JVgghKrlSJ9kRI0Zw4MABYmJiyMzMJC4ujpMnTzJixAjeeOMN8y1dIYQQQpiUOslGRkayZMkS6tevj7e3NxEREbzxxhv8+uuvrFmzhvvuu4/4+Hh79vWecfDEZZ6ft4uzaaWb9UoIIUTFVOpnsgCdO3emc+e/p/wbOHAg/fr1Iz8/H29v73Lv3L1KrVaRk1/Ipat5uLs646rV4ObihKIo5OsMqFUqVCpTu6I/y61lIYSoeMqUZME0If+VK1csBkHB3zM3Ffd+qygbTzdnABZ8cQiA2A516f9AKLkFhTw/739W7eOi6vNwp/qkX8snYcle6tX0onXjQFo3DqSar9td7bsQQoi/lTrJnj17loSEBPbt22dztico/YxPwraQml482zec3IJCjEaF4OqmqSCdNWqeeCAUBQWjUcGogGJUCKvrB4CLVkOXlrU5evoqa5JOsCbpBHVreDIkJoyGtX1K+kohhBB2UOok+/rrr3PgwAH69OlDUFAQGo3Gnv26p6lVKto2qW5VrnXW0KtDXZvbebg682SPxgBcvJrLL39e5uc/0/BxN436/vX4Zf48k0HrxoE0qO2NWm4xCyGEXZU6yf7yyy+MHj2aF1980Z79EeWkhp87vTrUtUjKKZey+GZfKlt+TMHHU0urRoG0blyNZvX85ZmuEELYQamTrI+PD35+fvbsi7CzRzrVp0ebIA7+dYWf/0xjz+/nOZ6awZujTK9e7fj1LIpRwclJjbOTGmeNhmo+roTUNK2IlHopGyeNCmfN9XonDVpnNU6aUg9SF0KIe0qpk+zgwYNZs2YNjzzyCL6+vnbskrAnd1dnIpvVJLJZTXR6A1eu5Zvrvth5kuw8vUX7juE1GfXQfQC8+dF+Cg2WA94eaF2HIQ+GYTQqTPtoH17uWrzdnU2/e2gJC/alYR0fjEaF9Gv5eHlocXGWRw1CiHtDqZPsyJEj+eGHH+jRowetW7cmICDAqo1KpeKtt94q1w4K+9E6a6gV4GH+/M7YSPQGI/pCA/pCI/pCI64uf58iYx9pht5gqiu8Xh90fVCWvtCIv5cr13J1XEzPJStXT4HeQN+o+jSs40Nmjo6XP/gBABdnDV7uzlT3d6d7qzq0ahx4dw9cCCHuklIn2Q8//JDvv/8egF27dhXbRpJs5ebuWvLp0CbMdjJ00Wp44fEIi7ICnQEF00h0V62Gp2ObcC1XR1aunmu5OjJz9BSNUz957hoffPk7oUE+hNYx/QoK9EStlmfFQojKq9RJdtWqVURERPD2229Tr149GV0sbslF+/c54ubiRFQLy3eoAwO9SEvLAkCjVhFSw4s/Tl1l7+GL5u0nP9makJpeZOfpUatUt/xBQAghKpJS/4+VlZXFuHHjaNiwoT37I+5RITW9GNevOYqicDkznxNnMzlxJpMa/qbJNL7Zl8qmPaeoE+hBg9o++Hpq8XBzpkebIFQqFZcz8ig0Kni4OuHu6oRGLYOxhBCOV+ok27JlS44ePWrPvgiBSqUi0NeNQF83IpvVNJe3alwNjVrFibOZ/PxnGtl5ely1Gnq2DQbg810n+fGPi+b2bi5O1PBz47Xh7QDY8mMKlzLyqOnnRkhNL4Kre8lVsRDC7kr9v8yrr77KsGHDcHd3p0ePHlSrVq3YW8YyraKwh3o1valX8+/5sQ1GI/k6g/nzg+2CiWgYQE6enpz8QnLy9BavFqVeyuLQyXSL0dPN6vszcUBLAJLPX8Pf2xUfD1muUQhRflRKSXMk3qBVq1bo9XoKCwtLnLigqk2reOVKNkZjqUJUrBufOwpLjohNZo6OlItZnL6QhYuzhp7tglEUhfELvudarh5fTy11a3gRUsOLZvX9aRzse1f7V0TOG9skNrZJbGwrj9io1SoCAjzLtE2pr2RHjBghswKJSs/HQ0vzBgE0b2D5CtozfcM5fSGL0xezSbmYxaGTVyjQG2gc7Iu+0Mi8dQep4e9OrQB3agd4UCvAHT8vF/k3IYQoUamT7PPPP2/PfgjhMCqVirC6fuaFFgAK9Kb3gQGycnXoCg389MdFcgsKzW0GdW9Ez3bBZObo2H3ovDkBB/q6yatHQgighCR77tw5/P39cXV1NX8uDXkmK6oCF2eNeWYqf29X/jmkLYqicC1Hx7kruZy/kkPY9VvJZy5l898df5m3ddKoqOHvzrBeTQit48Nf5zL54fcL11dOUjAYTaso9evSkAAfV3776zI7fjlnUefq4sSgbqFU83UjM0eH0ajg66mVK2chKhmbSbZ79+7MmjWLhx9+GIBu3bqV6h94VXsmK0QRlUqFj6cLPp4uNA35+6q3WX1/Fr4UxfkruZy7ksP5K7mcv5yDx/XRy1cy8/npyCXUKtMzHY1ahVqtIl9vGriVrzOQnpWPWmWqU6lV6PP0uF3fPunAGTbuOYWrVkOtAHdq+ptuV8e0r4uzkxpFUST5ClFB2Uyy48aNIywszOKz/EMWonjurs40rONDwzrW6/a2b1qD9k1r2Ny2uPobB2m0a1IdH0+tKXlfyeFoylV+Pp5Gn8gQAFZtPcaxlAxTAg5wx8ddi5+3K+2uL5eYmaPDSaPCzcWpQixvWHD9hwsXZw36QiNZuTrcXJxw0WoqRP+EKE82k+xzzz1n8bk0z2R1Ot2d90gIYSGouqd5jugiOr3B/ENvvZpeZOfqOZ+ey29/XcFgVKjp725Osh9s+J1jqRmoVKY1hz3dnGlY25uR1xd++PbAGfJ1hXi5a/F0M9X7e7lQzdfttvqrXL/t7aRRk52n59sDZ0jLyONSRh5pV/PIzNExPLYJXVrU5kxaNm9+tB8AlQrctKbJRJ7s2ZiWodU4m5ZN4o8puLk44e7ihJuLE4EBHjSo4YmflwuXMvI4evqqVR9ahlbD20PLhfRc/kzNMJdr1CqcndSE1/fH3dWZq1kFXMnMx8nJtLqUk5MaZ40aH08tGrUao1FBpUIuMMRtK/XAp+7du5OQkED37t2Lrd+0aRNvvvkmP/74Y7l1TghRPO0NKxl1bVmHri3rAGBUFPIKCim44R3imPZ1adWoGtn5erLzCsnO1eHh5myu3/HrWc6m5VjsP7yBPxOeaAnAG8t/QldoxNPdGS83Z7zcnWkc7EvH8FoUGozsOnjOlESv5pGWkU9aZh4x7YLpG9UAo6Lw5ffJ+Hm5EOjrRvMGAQT6uVHv+vKJAd6uDI9tQm5+IXkFheQWFJKbX2h+Xzk7T8+fqRnm+qKX6SYOaImflwunL2SxMtF6kpx/DmmDt4eW42cyiq1/c2R73F2d2X/0Ep9+e9yq/t1nOhLg48rXP5xi/f+SUatUqNWgVqlQqVT8a1xH3F2d+er7ZJJ+PoNKrTK1ud7urTH3o1Gr+Wp3MvuOXDInapUKtE4aEoa0AeDL75P5PfkKKpUKNaY2Hm7OPNevOQDrd53kxNlMi775emoZ/XAzAD5LOsHpi3+/lqLVavD3dGFITJh5/1eu5eOsUaPRqHDSqKnu50b09fPlh98vkK8rRKNRo3VSo3XW4OflQv1apnfSL2XkoVGp0Dqb6pyd1HK3oYxsJtn09HT++uvvwRxnz57l0KFDeHt7W7U1Go188803ciUrhIOpVSo8XJ3xcP07ibZsVK3Ebd4c2YECvYGcPD1ZuXqy8/QWyxE2CfEjPauA7FwdaRl5nDx/DbVaTcfwWmjUKtYmnQAwzdTl40qTEF9Cg0y3zb3cnFk8qSvOTsXPde7toaVLC9uDJcPq+jHrmY6A6QeIAp0BDy9XCnILAIhoGMDsZztabeflbkrS7ZvUoFk9f3N5ocGI3qBQ/fpVeuvGgdQKcEdvMFJoUEyrSxmMeF7/IaRxsC+PdKpnGpCmKChGUz+KJjqpE+hJ68aBGBUF4/U6RVHMicjbQ0tNf3cUTFf4imK6mi7i7KTGxVmDovxdf+N7+Qajgv6m5SX1hr/rTcdzQ71eRb7u7xHwptfSsig0GK//UmhY29ucZL/ac4qL6bkW+49oGMBL/VsA8M7qA2RkW/6/fn+zGoy5nuSnrdyH0aiYE7CLs4bmDQN4oFUdFEVh3Xd/4eSkxsVZjdZJg7OzmpAaXtSv5Y3RqHD8TAauWie0zqZtXbUaXLSaKjUtqs3JKHJycoiNjSUtLa1UO1IUhd69ezNnzpxy7aCjyWQU9iOxsa0yxeZajg5Pd+e7doVTmWJzt5U1Nrn5haYfMAqN6AoN6PRGtM5q8xKY+49eIregEJ3egK7QiE5voHY1D/MYgiUbD5OXX4ju+tKXOr2B1mGBPNKpPoUGI8/N24Veb+TG/0FjO9Sl/wOh5OTreX7e/6z61LdzfR7pXJ+rWQVMX7XfPNLfRWv6vVvrOrQIrcbVrAK+/P4kGo0aJ7UaJ40KjUZNm8aBhNT0IjO7gP3H0kxX8Go1bcNr4XqHubtcJ6Pw8PDg/fff588//0RRFBISEnjiiSdo1apVMV+sxt/fn8jIyLL3WghRqXnLVJSV1q3m7257/bm+LUVXtMVx0qj5YGI0iqJQaDBSoDclYmcnU6ZzcdYwcWBLdDoDBXoD+XoDOp3BPHhQo1bRrJ4/BXpTfYHOQFauzjydam6+noN/XcFgUMxX6QaDkRrX5ye/eDWPT77509wfrasz7RuXfFfHHko9reLChQt58MEHady4sb37VKHIlaz9SGxsk9jYJrGxTWKD+ZW2QoOR3IJCDNeTb90gP3Kz8+9o33adVvHm0cYAer2e3bt3m57PdOyIk5OsaiKEEMJxikaCO2nUeLv/fZfFw835jpPs7Sh1VtTpdEyfPp0zZ86wfPlydDodAwYMMC9/17BhQz766CMCAgJusSdLmzZt4v333yc1NZU6deowduxY+vbtW6ptZ86cyZEjR1i5cqVF+f79+xk8eLBV++joaBYvXlym/gkhhBC3q9RJduHChXz22Wc89thjAGzYsIEjR44wdOhQmjZtyjvvvMP8+fOZNm1aqb88MTGRSZMmMXToUKKioti+fTuvvPIKrq6u9OrVq8RtV69ezfLly4t9Dnzs2DHc3d1ZsWKFRXlxI6OFEEIIeyl1kk1MTOTxxx9n+vTpAGzduhUvLy9efvllnJycSE1NZd26dWX68jlz5hAbG0tCQgIAUVFRZGZmMn/+fJtJ9uLFi8yaNYvNmzfj5eVVbJujR4/SqFEjWrZsWab+CCGEEOWp1AOaL1y4YE5aeXl57Nu3j8jISPNz2Fq1anHt2rVSf3FqaiopKSk8+OCDFuUxMTGcPHmS1NTUYrebO3cuf/zxBytWrKBp06bFtjly5IjFlJBCCCGEI5Q6yVarVo3Lly8D8L///Q+dTkd0dLS5/tixY1SvXvJw7xudPHkSgPr161uUh4SY5mNNTk4udrtRo0bx9ddfc//99xdbbzQaOX78OBcuXCAuLo7w8HCio6NZvnw5pRxILYQQQpSLUt8u7tChAx999BEuLi588sknuLm50aNHD65du8bnn3/OZ599xsCBA0v9xVlZpmHmnp6Ww6E9PEwvQWdnZxe7XWhoaIn7TU5OJj8/n+TkZCZMmICfnx/ffvsts2bNIjs7mxdeeKHUfRRCCCHuRKmTbEJCAhcvXmTmzJm4u7szffp0vL29OXDgADNnzqRdu3bFvuZjS9FV5c0TbxeVq29zWq0aNWqwdOlSmjZtSmBgIACRkZHk5+ezdOlSRowYYZXYS1LWd6KKExhY/LNjIbEpicTGNomNbRIb2xwRm1InWW9vb1asWEF6ejqenp5otab3j5o2bcratWtp0aJFmb64aNDSzVesOTk5FvVl5enpSZcuXazKo6OjWbduHcnJyTRv3rzU+5PJKOxHYmObxMY2iY1tEhvbyiM2tzMZRZkvF/39/c0JFsDd3b3MCRb+fhabkpJiUX769GmL+rI6duwY//nPf9Dr9Rbl+fmml5D9/PyK20wIIYQodzaT7HPPPcf+/fstyoxGI0ePHiUvL8+q/VdffWVztG9xQkJCCAoKYsuWLRbl27Zto169etSubXtljpKcPn2aqVOnsmvXLovyzZs3ExQURJ06dW5rv0IIIURZ2bxdvH37dmJiYizKMjMziYuLszkJRFmNGzeO+Ph4fHx8iI6OJikpicTERObOnQuYlttLSUkhNDS01M9Ro6OjCQ8PZ8qUKaSnp1OzZk02btxIUlISCxYskMWXhRBC3DVlnmy4PF+D6devHzqdjuXLl7Nu3TqCg4OZOXMmvXv3BmDHjh3Ex8ezatUqOnToUKp9arVali5dyrx581i4cCHp6ek0atSIhQsX0qNHj3LruxBCCHErDp/Rf+DAgTZf/enXrx/9+vWzue3HH39cbLm/v3+ZpncUQggh7KHqLD8vhBBCVDCSZIUQQgg7kSQrhBBC2EmJz2QzMjI4d+6c+XNmZiZgGvV7YznA1atX7dA9IYQQovJSKTaGCzdp0qTY110URSnxNZgjR46UX+8qAJnxyX4kNrZJbGyT2NgmsbHNUTM+2bySjYuLu6POCCGEEPc6m0n27bffvpv9EEIIIaocGfgkhBBC2IkkWSGEEMJOJMkKIYQQdiJJVgghhLATSbJCCCGEnZRbkjUYDPz73/8ur90JIYQQlV6JMz5du3aNdevWcfDgQRRF4b777mPw4MF4e3tbtDt48CCvvvoqJ06c4Nlnn7Vrh4UQQojKwmaSTUlJYciQIVy6dMm8huw333zDJ598wmeffUbt2rXR6XTMnj2bTz75BIPBwMMPP3zXOi6EEEJUdDZvF8+ZM4dLly4xceJEdu/ezc8//8y8efMwGo1MmzaNa9eu8dRTT7Fq1Spq167NsmXLePfdd+9m34UQQogKzeaV7IEDB4iLi2PUqFHmsl69elFQUMA///lP/vGPf3Do0CGGDh3KxIkTcXFxuSsdFkIIISoLm0n26tWrtGrVyqq8bdu2FBYW8uOPP7Jo0SK6detm1w4KIYQQlZXN28WFhYW4urpalbu7uwMwcuRISbBCCCFECW77FZ7WrVuXZz+EEEKIKue2k6xaLfNYCCGEECUp8T3ZkydPsm/fPouyrCzTorfHjh3Dycl683bt2pVj94QQQojKS6UUvQR7kyZNmqBSqYrdSFEUm3VHjhwpv95VAFeuZGM0FhuiUgkM9CItLasce1R1SGxsk9jYJrGxTWJjW3nERq1WERDgWaZtbF7JPvfcc3fUGSGEEOJeJ0lWCCGEsJNyG72Un5/PuXPnymt3QgghRKVnM8nGxcWxa9cuizK9Xs+3337L1atXrdpv27aN7t27l38PhRBCiErKZpI9cuQImZmZFmXZ2dk899xzHD161O4dE0IIISq7Mt8utjEYWQghhBA3kRklhBBCCDtxeJLdtGkTffr0ISIigtjYWDZs2FDqbWfOnMnw4cOtygsLC5k3bx5du3alRYsWPPnkk/z222/l12khhBCiFByaZBMTE5k0aRKdOnVi0aJFtG/fnldeeYUtW7bcctvVq1ezfPnyYutmzJjBypUrGT16NHPnzkWj0TB8+HBSU1PL+xCEEEIIm0qcVtHe5syZQ2xsLAkJCQBERUWRmZnJ/Pnz6dWrV7HbXLx4kVmzZrF582a8vLys6s+cOcPatWuZMmUKgwYNAqBz587ExMTw4YcfMnXqVPsdkBBCCHGDMs1dXNK8xX/99VeZvjg1NZWUlBQmTJhgUR4TE0NiYiKpqakEBwdbbTd37lz++OMPVqxYwaJFi6zq9+7di8FgICYmxlym1WqJjo5mx44dZeqjEEIIcSdKTLIffPABH3zwgVX5zJkzrcpKms+4OCdPngSgfv36FuUhISEAJCcnF5tkR40aRYMGDVCr1cUm2ZMnT+Lj44O/v7/Vfs+dO0d+fn6x6+TaS3Q06PVuFmWPPFLIiBF6cnPhySfdrLYZOFDPwIGFXLmiYuRI674OH66nb99Czp5VMW6cdf0zz+iIiTFw4oSKSZOs68eP19G1q4FDh9RMmeJiVZ+QUED79kZ++knNW29Z17/5ZgHNmxvZuVPD3Llaq/rZs/MJDVXYulXD++9b1y9alE+dOgpr18J771kf/7Jl+QQEKKxZ48SaNc5W9f/5Tx7u7rB8uTNffWV9Cm/YkHf9e5z55hvLeldXWLPGVP+vf2n53/80FvV+fgorVuQDMH26lv37Letr1VJ4/31T/auvuvD775ZPXBo2NPKvfxUAMHGiC3/9ZVkfHm5k+nRT/TPPuHL+vOW/mbZtDbz6qg6Ap5925epVy/qoKAMTJ5rqBw50Iz/f8th79ixk3Dg9AH37Wse2Kpx73bpxx+fehg1OrFxpfW5V9nNv1SrTn+Xcs67fvduq6K5w2LSKRVfFnp6Wky17eHgApndyixMaGlrifrOzs632eeN+c3JyypRkyzoZdHGcnS3D7OXlRGCgK7m54Gz97/h6PahUxdd7e5vq8/OLr/fxMdWnpxdf7+trqvf3L77ez89U7+dXfL2/v6ne19dWvSeBgeDjU3x9QICpHqxjA1CtmifVqoGXV/HbBwZ64e5ecj2Ap6d1vVb7d72Hh3W9iwsEBpoK3d2t611d/653c7NVrzX/+eZ6N7eS693dnQgMdLneF2ereg+Pv+u1WjAYLOs9PU3nFpR0blXuc8+0H/c7Ove8vYuvr+znXlEf7vzcs66v7Oce/B3/u8nmKjz2tnHjRiZNmkRSUhJ16tQxl586dYqYmJgSn8sWGTJkCBqNhpUrV5rLpkyZwu7du0lKSrJo+9lnnzFlyhR++OEHq6vcksgqPPYjsbFNYmObxMY2iY1tjlqF545GF2dlZdm84ryVokFLN2+fk5NjUV9Wnp6e5n0Ut9/irnKFEEIIeygxySqKws6dO1m2bBnbtm2jsLAQgB9++IE+ffrQvn172rVrR79+/fj+++/L9MVFz2JTUlIsyk+fPm1RX1YNGjQgIyPDakrI06dPExQUhFZr/ZxGCCGEsAebz2SvXbvGmDFjOHjwoHkqxfDwcF577TXGjBmDm5sbPXr0wGg0snfvXsaOHcuKFSto3759qb44JCSEoKAgtmzZQs+ePc3l27Zto169etSuXfu2Dqhjx44AbN26lSeeeAIAnU7Hzp076dy5823tUwghhLgdNpPse++9x9GjR3nttdfo0KED58+fZ8aMGQwbNox69erx8ccf4+vrC8Dly5d54oknWL58eamTLMC4ceOIj4/Hx8eH6OhokpKSSExMZO7cuQCkp6eTkpJCaGhoqW/z1qlTh7i4OKZPn05ubi4hISGsWLGCzMxMRo0aVeq+CSGEEHfKZpJNSkpi4MCB5gkdGjRowKuvvsqIESMYPHiwOcECVKtWjSeeeIKPP/64TF/er18/dDody5cvZ926dQQHBzNz5kx69+4NwI4dO4iPj2fVqlV06NCh1PudNm0a3t7eLFmyhNzcXJo1a8aKFSvMrwcJIYQQd4PNJJuWlkbDhg0tyopenynuVm6tWrWsnoOWxsCBAxk4cGCxdf369aNfv342t7WV1LVaLQkJCeaZpIQQQghHsDnwSa/XW71P6nz95SPnYl5CUqlUGG5+cUoIIYS4hzl8FR4hhBCiqipxWsWMjAzOnTtn/lx0Ozg9Pd2iHODq1at26J4QQghRedmc8alJkybFzkV8qzmKjxw5Un69qwBkxif7kdjYJrGxTWJjm8TGNkfN+GTzSjYuLu6OOiOEEELc62wm2bfffvtu9kMIIYSocmTgkxBCCGEnkmSFEEIIO5EkK4QQQtiJJFkhhBDCTiTJCiGEEHYiSVYIIYSwE0myQgghhJ1IkhVCCCHsRJKsEEIIYSeSZIUQQgg7kSQrhBBC2IkkWSGEEMJOJMkKIYQQdiJJVgghhLATSbJCCCGEnUiSFUIIIexEkqwQQghhJ5JkhRBCCDuRJCuEEELYiSRZIYQQwk4kyQohhBB2IklWCCGEsBNJskIIIYSdODzJbtq0iT59+hAREUFsbCwbNmwosX1OTg5Tp06lU6dOtGrVitGjR3Pq1CmLNvv37ycsLMzq19ixY+13IEIIIcRNnBz55YmJiUyaNImhQ4cSFRXF9u3beeWVV3B1daVXr17FbjN+/HgOHTrEyy+/jIeHBwsXLmTo0KF8/fXXeHl5AXDs2DHc3d1ZsWKFxbbe3t52PyYhhBCiiEOT7Jw5c4iNjSUhIQGAqKgoMjMzmT9/frFJdv/+/ezcuZOlS5fSpUsXANq2bUv37t359NNPGTNmDABHjx6lUaNGtGzZ8q4dixBCCHEzh90uTk1NJSUlhQcffNCiPCYmhpMnT5Kammq1ze7du/Hw8KBTp07mMn9/f9q1a8euXbvMZUeOHCEsLMx+nRdCCCFKwWFJ9uTJkwDUr1/fojwkJASA5OTkYrcJCQlBo9FYlNetW9fc3mg0cvz4cS5cuEBcXBzh4eFER0ezfPlyFEWxx6EIIYQQxXLY7eKsrCwAPD09Lco9PDwAyM7OttomOzvbqn3RNkXtk5OTyc/PJzk5mQkTJuDn58e3337LrFmzyM7O5oUXXijvQxFCCCGK5bAkW3RVqVKpii1Xq60vsku6Ei1qX6NGDZYuXUrTpk0JDAwEIDIykvz8fJYuXcqIESOKTdS2BASUvq0tgYFed7yPqkpiY5vExjaJjW0SG9scERuHJdmikcA3X7Hm5ORY1N/I09OTM2fOWJXn5OSYE6enp6d5UNSNoqOjWbduHcnJyTRv3rzU/bxyJRuj8fZvMwcGepGWlnXb21dlEhvbJDa2SWxsk9jYVh6xUatVZb7wctgz2aJnsSkpKRblp0+ftqi/eZvU1FSrK9rTp0+b2x87doz//Oc/6PV6izb5+fkA+Pn5lc8BCCGEELfgsCQbEhJCUFAQW7ZssSjftm0b9erVo3bt2lbbdO7cmWvXrrFnzx5zWXp6Ovv376djx46AKeFOnTrVYrQxwObNmwkKCqJOnTp2OBohhBDCmkPfkx03bhzx8fH4+PgQHR1NUlISiYmJzJ07FzAl0JSUFEJDQ/H09KRdu3a0b9+eCRMmMGnSJHx9fVmwYAFeXl4MGjQIMN0WDg8PZ8qUKaSnp1OzZk02btxIUlISCxYssHoGLIQQQtiLSnHwey1r1qxh+fLlnD9/nuDgYMaMGUPfvn0B+OKLL4iPj2fVqlV06NABgMzMTN555x22b9+O0WikTZs2TJ48mQYNGpj3mZ6ezrx589i5cyfp6ek0atSIZ599lh49epS5f/JM1n4kNrZJbGyT2NgmsbHNUc9kHZ5kKzpJsvYjsbFNYmObxMY2iY1t99zAJyGEEKKqkyQrhBBC2IkkWSGEEMJOJMkKIYQQdiJJVgghhLATSbJCCCGEnUiSFUIIIexEkqwQQghhJ5JkhRBCCDuRJCuEEELYiSRZIYQQwk4kyQohhBB2IklWCCGEsBNJskIIIYSdSJIVQggh7ESSrBBCCGEnkmSFEEIIO5EkK4QQQtiJJFkhhBDCTiTJCiGEEHYiSVYIIYSwE0myQgghhJ1IkhVCCCHsRJKsEEIIYSeSZIUQQgg7kSQrhBBC2IkkWSGEEMJOJMkKIYQQdiJJVgghhLATSbJCCCGEnTg8yW7atIk+ffoQERFBbGwsGzZsKLF9Tk4OU6dOpVOnTrRq1YrRo0dz6tQpizaFhYXMmzePrl270qJFC5588kl+++03+x2EEEIIUQyHJtnExEQmTZpEp06dWLRoEe3bt+eVV15hy5YtNrcZP348W7ZsYdKkScycOZOLFy8ydOhQsrKyzG1mzJjBypUrGT16NHPnzkWj0TB8+HBSU1PvxmEJIYQQADg58svnzJlDbGwsCQkJAERFRZGZmcn8+fPp1auXVfv9+/ezc+dOli5dSpcuXQBo27Yt3bt359NPP2XMmDGcOXOGtWvXMmXKFAYNGgRA586diYmJ4cMPP2Tq1Kl37wCFEELc0xx2JZuamkpKSgoPPvigRXlMTAwnT54s9qpz9+7deHh40KlTJ3OZv78/7dq1Y9euXQDs3bsXg8FATEyMuY1WqyU6OtrcRgghhLgbHHYle/LkSQDq169vUR4SEgJAcnIywcHBVtuEhISg0WgsyuvWrUtiYqK5jY+PD/7+/lb7PXfuHPn5+bi6upa6n2q1qtRt7bmPqkpiY5vExjaJjW0SG9vuNDa3s73DkmzRM1RPT0+Lcg8PDwCys7OttsnOzrZqX7RNUfuS2oBp4FRZkqyfn0ep29oSEGDdH2EisbFNYmObxMY2iY1tjoiNw24XK4oCgEqlKrZcrbbuWlFdcYra22pj6/uEEEIIe3FYkvXy8gKsr1hzcnIs6m/k6elprr95m6Kr15LaFNULIYQQd4PDkmzRs9iUlBSL8tOnT1vU37xNamqq1dXq6dOnze0bNGhARkYGmZmZVm2CgoLQarXldgxCCCFESRyWZENCQggKCrJ6J3bbtm3Uq1eP2rVrW23TuXNnrl27xp49e8xl6enp7N+/n44dOwKYf9+6dau5jU6nY+fOneY6IYQQ4m5w6Huy48aNIz4+Hh8fH6Kjo0lKSiIxMZG5c+cCpgSakpJCaGgonp6etGvXjvbt2zNhwgQmTZqEr68vCxYswMvLy/xObJ06dYiLi2P69Onk5uYSEhLCihUryMzMZNSoUY48XCGEEPcYlVLSaKK7YM2aNSxfvpzz588THBzMmDFj6Nu3LwBffPEF8fHxrFq1ig4dOgCQmZnJO++8w/bt2zEajbRp04bJkyfToEED8z51Oh2zZ89m06ZN5Obm0qxZM15++WVatGjhiEMUQghxj3J4khVCCCGqKocvECCEEEJUVZJkhRBCCDuRJCuEEELYiSRZOynrOrmVUWFhIREREYSFhVn8atWqlbnN999/z2OPPUaLFi3o1q0by5cvt9rPoUOHGDJkCK1ataJz587MmTMHvV5v0ebUqVP83//9H23btqVDhw68/vrrxU696WhHjhyhWbNmXLhwwaL8bsbh8uXLTJw4kQ4dOtCmTRsmTJhAWlpa+R9sGdmKTc+ePa3OobCwMNLT081tqmJsjEYjn376KQ8//DCtWrWiR48evP322xZ9vlfPm9LEptKcN4ood5s3b1bCwsKUGTNmKLt27VJee+01pXHjxkpiYqKju1au/vzzT6Vx48bK+vXrlV9++cX86+DBg4qiKMqBAweUZs2aKZMmTVJ27typzJkzRwkLC1M+/PBD8z5OnTqltG7dWhk5cqSyY8cOZdmyZUp4eLgydepUc5uMjAylS5cuymOPPaZs375dWbt2rdK2bVtlzJgxd/2YS/LXX38pUVFRSuPGjZXz58+by+9mHPR6vfLII48oPXr0UDZv3qx89dVXSteuXZVHHnlE0ev1dycQxbAVm+zsbCUsLExZvHixxTn0yy+/mPtbVWOzePFipWnTpsrs2bOV3bt3K6tXr1bat2+vjBgxQlGUe/u8uVVsKtN5I0nWDnr06KG89NJLFmUvvvii0qtXLwf1yD6++uorpUmTJkpubm6x9cOGDVP69+9vUTZr1iylbdu2SkFBgaIoipKQkKB07drV/FlRFOWTTz5RmjZtqly4cEFRFEVZtGiR0rJlSyU9Pd3cZseOHUrjxo2VX3/9tbwPq8z0er2yevVqpVWrVkr79u2tEsndjMOGDRuUxo0bKydOnDC3OX78uBIWFqZ8/fXX5X/wt3Cr2Bw4cMCqvzerirExGo1Ku3btlDfeeMOi/Ouvv1YaN26s/PHHH/fseVOa2FSm80ZuF5ez21knt7I6cuQIdevWxc3NzaquoKCA/fv3FxuHa9eu8fPPPwOmNYIfeOABi+kue/XqhcFg4Pvvvze3adeuHX5+fuY2nTt3xsPDg507d9rj0MrkwIEDzJ49mxEjRjBp0iSLursdh927dxMaGkrDhg3NbYo+OyJWJcUGTOeQi4sL9erVs7mPqhibnJwcHnnkER566CGL8qL3/Y8fP37Pnje3ik1KSkqlOm8kyZaz0qyTW1UcO3YMrVbLyJEjadWqFe3ateO1114jOzub1NRU9Hp9iXHIy8vj/PnzVm38/f3x9PQ0x+rkyZNWbTQaDUFBQRUing0bNmT79u0899xzVmsd3+04FNcGTGsuOyJWJcUGTOeQr68vEyZMoG3btrRq1Yrx48ebn3dV1dh4enry6quv0qZNG4vy7du3A3Dffffds+fNrWITGhpaqc4bSbLl7HbWya2sjh49SkpKCl27dmXJkiU8++yzbNq0iWeeeaZUcbDVpqhdUayysrJu2caRqlWrRkBAQLF1dzsOFS1WJcUGTOfQ5cuXadSoER988AHx8fHs27ePoUOHkp+fX6Vjc7ODBw+yZMkSevTocc+fNze7MTYNGzasVOeNQ+curoqU21gnt7KaO3cuPj4+hIWFAdCuXTsCAgL4xz/+we7duwHb6/eq1eoS1/hVFMUiVqVpUxGVdIxgnzhUpli9+uqrKIpinvK0bdu2NGzYkCeffJKvvvqKrl27AlU/NgcOHOD//u//CAoKYvr06eYrJDlvrGMDleu8qXj/6iq521knt7Jq3769OcEWiY6Otvh8cxyKPnt5eZl/Oizup8Hc3FxzrDw9PYttc+M6whWVrfPBXnGobLGKiIiwmlO8TZs2eHl5cfTo0XsiNps3b+bpp5+mVq1arFy5Ej8/PzlvrisuNlC5zhtJsuXsdtbJrYyuXLnCunXrrAZy5efnAxAQEIBGo7GKQ9Hn+vXr4+HhQY0aNcyxuXHf2dnZ5ljVr1/fqo3BYODMmTMVPp5169a9q3Eork3R91W0WOXm5vL5559z9OhRi3JFUdDr9fj5+VX52KxYsYIJEybQsmVLPvnkE6pXrw7IeQO2Y1PZzhtJsuXsdtbJrYxUKhWvvfYaq1evtijfvHkzGo2Gjh070rZtW7Zt22a+rQWmdX69vLwIDw8HoFOnTnz33XfodDqLNhqNhvbt25vb/Pjjj2RkZJjbfP/99+Tm5lb4NYJdXFzuahw6d+7M8ePHzQPwAE6cOMHJkycrXKxcXFyYOXMmCxcutCj/9ttvyc/PtzjuqhibdevW8c477xAbG8uHH35ocZfrXj9vbhWbSnXe3PIlH1Fmn3/+udK4cWNl6tSpys6dO5XXX39dady4sUPeU7SnN998U2natKny3nvvKXv27FEWLFigNGvWTJk+fbqiKIqyZ88eJSwsTHnxxReVHTt2KHPnzlXCwsKUJUuWmPdx4sQJpXnz5sqwYcOUpKQkZfny5Up4eLjy+uuvm9tcuXJF6dChg/Loo48q27ZtUz777DOlXbt2yqhRo+72Id9S0d/9je+C3s04FBQUKDExMcoDDzygbNq0Sdm4caMSHR2tPPTQQw6djEJRio/N8uXLlcaNGytvvvmmsnv3bmXFihVK69atlWeeecbcpirG5vLly0qLFi2UBx54QNm3b5/VhApXrly5Z8+b0sSmMp03kmTt5NNPP1V69uyphIeHK7Gxscr69esd3aVyp9PplCVLligxMTFKeHi40r17d2Xx4sWKwWAwt9m2bZvy0EMPKc2aNVO6deumLFu2zGo/+/btU/r376+Eh4crUVFRyr/+9S9Fp9NZtDl27JgybNgwJSIiQomMjFSmTJmiZGVl2f0Yy6q4RKIodzcO586dU8aNG6e0bNlSad++vTJ+/Hjl4sWL5X+wZWQrNp999pny0EMPKREREUpUVJQya9YsJS8vz6JNVYvN+vXrlcaNG9v8tWHDBkVR7s3zprSxqSznjawnK4QQQtiJPJMVQggh7ESSrBBCCGEnkmSFEEIIO5EkK4QQQtiJJFkhhBDCTiTJCiGEEHYiSVaIOzB58mTCwsJu+Wvy5Ml3/F1ffPEFYWFh/Pjjj+XQc5NDhw7xwgsv0KlTJ8LDw+ncuTPjx4/n0KFD5fYd5c0ecRDCXmQVHiHuwIABA4iMjDR/PnDgAGvXrmXAgAEW62HWrVv3jr+rXbt2zJo1y2Lx6Duxa9cu/u///o/Q0FCGDh2Kv78/58+f5/PPP2fLli0sWLCAHj16lMt3CXGvkiQrxB1o1aoVrVq1Mn82GAysXbuWli1b8uijj5brdwUHBxMcHFxu+5sxYwZNmjRh7dq1ODs7m8uHDh3Ko48+yrRp04iOjsbJSf6bEOJ2ye1iIe5B6enpnDp1ig4dOlgkWABfX1/69u3L5cuXOXPmjIN6KETVIElWiLtkwYIFNG/enG+++YZOnTrRqlUr1q1bB8Dhw4d5/vnn6dixI82aNSMyMpKJEydy4cIF8/Y3P4ss+nz06FEmTpxIu3btaNWqFePGjbtlcnRzc0Oj0fDtt9+SlpZmVf/8889z+PBh6tWrZy5LS0tj2rRpdO/enfDwcNq0acPQoUM5cOCAuc2PP/5IWFgYe/bs4Z///Cft2rWjTZs2xMfHk5uby86dO3n00Udp0aIFjz76KD/88IPV8f32228888wztGzZks6dO/P222+bl1C0paCggLlz59KtWzfCw8Pp3r078+fPt1iBBUyrsDz22GO0atWKNm3a8PTTT1v0X4jyJveBhLiLCgsLefXVVxk5ciQ6nY42bdpw7NgxnnzySUJCQhgzZgxubm78/PPPfPnll1y6dImPP/64xH0+88wzNGzYkPHjx5OamspHH33ExYsX+e9//2tzGzc3N3r37s3GjRvp0aMH3bp1o3Pnztx///3UqVPH6hZxfn4+gwcPJisri8GDB1OjRg1OnTrFp59+ypgxY9i5c6fFAtaTJ08mNDSUiRMn8tNPP/HFF19w4cIF/vjjD4YMGYKXlxdLlizhxRdfZPv27Xh7e5u3ffHFF6levToTJ07kyJEjrFy5khMnTrBs2bJij8VgMDB27Fh+/vlnnnjiCRo2bMjvv//OBx98wJEjR3j//fdRqVT89NNPjB8/ni5dutC/f3/y8vJYvXo1Tz/9NF9//XW53ooXoogkWSHuIqPRyFNPPcWYMWPMZa+//joqlYpVq1bh6+sLmAZU6fV6vv76azIyMszlxQkPD2fBggXmz7m5uaxZs4ZTp05ZXInebOrUqRgMBjZv3mz+BdCoUSMGDRrEoEGDUKtNN7uSkpI4ffo0H374IVFRUeZ9BAcH8/rrr7Nnzx4efPBBc3n16tX58MMPUavVPPHEE/z000/s2bOHpUuX0qVLFwDc3d159dVXOXToEJ06dTJvGxAQwMcff4xWqwUgMDCQDz74gP/9738W313kyy+/5IcffrDqW0REBK+99hrffvstPXr0YPPmzbi6upqTLkDHjh154YUXOHz4sCRZYRdyu1iIu6xz584Wn9944w2SkpIsEml2djYuLi6AKWmWJDY21uJz06ZNAbh8+XKJ23l4eDB37lw2b97M888/T6tWrXBycuL48eNMmzaNZ599FoPBAEDv3r354YcfLPp+463Ym/vYvXt3c4JWq9UEBwfj6upqTrAAQUFBAFa3q0eMGGFOsABPP/00YEr0xdm2bRv+/v40a9aM9PR086+uXbui0WjYsWMHADVr1iQnJ4fp06fz119/ARAWFsbWrVvp1atXibES4nbJlawQd1lAQIDFZ5VKxdWrV1m8eDHHjh0jJSWFc+fOUbQKpdFoLHF/fn5+Fp+LElRRgryVhg0b8txzz/Hcc8+RlZXFli1bmD9/Pt999x1bt26ld+/e5n4uWbKEX375hZSUFFJSUtDr9cX2sVq1ahafnZyc8Pf3tygrSsI3b3vzK0q+vr74+vpy9uzZYvufkpJCenq6xatUNzp//jwATz31FN9//z2rV69m9erVBAUF8cADD/D444/TpEkTm/ER4k5IkhXiLitKLkV27NjBs88+S/Xq1bn//vvp0qUL4eHhfP/99yxevLjM+yuNHTt2sHv3bv7xj39YXDV6eXnRv39/GjduzBNPPMGBAwfo3bs3Z8+eZcCAAeTm5tK5c2d69+5N06ZNURSFcePGWe1fo9FYlRXdor2Vm0c7g+kHBlvHaTAYqFevHq+//nqx9UXPez09PVm9ejW//vor27dvZ9euXXz88cd88sknzJo1i4cffrhU/ROiLCTJCuFgb775JiEhIXz++ee4u7ubyzdu3Gi37zx8+DCrVq2iZ8+etG/f3qq+UaNGALi6ugKwcOFCrly5QmJiosVzXnv0MTU1lQYNGpg/p6enk5WVZfP5clBQEL///jv333+/RSLW6/V888031KxZE4Dk5GSysrJo2bIlLVu2ZNKkSZw4cYLBgwezYsUKSbLCLuSZrBAOlpGRQe3atS0S7Pnz59m2bRtQ+tu+ZdGnTx/UajUzZ87k2rVrVvVr164FTM9Wi/ro5uZG7dq1zW10Oh1r1qwp9z6uXr3afKscMI8q7tmzZ7Htu3XrRkZGBp9++qlF+Zo1axg/frz5NaHp06fz7LPPkpOTY27ToEEDvL29b+tugBClIVeyQjhYly5d2Lx5M6+99hrNmzfnzJkzfPbZZ+Tl5QFYJIXyUq9ePeLj43nrrbeIjY3lkUceoUGDBuTn57N7926+++47hgwZQuvWrc19TEpKYuzYsfTq1YusrCw2bNhASkpKuffxxx9/ZPTo0TzwwAMcPHiQL7/8kr59+1pMU3mj/v37s379et58800OHz5MREQEf/75J2vXrqVZs2b069cPMA2gGj16NIMHD6Zv3764uLiwfft2UlJSmDlzZrn1X4gbSZIVwsHeeOMN3N3dSUpK4ssvv6RmzZr07duXnj17MmjQIPbu3ct9991X7t87dOhQ7rvvPj755BM2b95Meno6rq6uNGnShDlz5tCnTx9z24EDB3Lt2jXWrVvH9OnTqVatGi1btmThwoUMHDiQvXv3Mnz48HLp11tvvcX69euZOXMmgYGBTJw4kVGjRtlsr9VqWblyJYsWLWLr1q189dVXVK9enUGDBjFu3Djc3NwA06ju999/n8WLF/Pvf/+bgoICGjVqZHWsQpQnlXLjfRkhhHCQL774gvj4eFatWkWHDh0c3R0hyoU8iBBCCCHsRJKsEEIIYSeSZIUQQgg7kWeyQgghhJ3IlawQQghhJ5JkhRBCCDuRJCuEEELYiSRZIYQQwk4kyQohhBB28v9m9q3jNRpy9gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 504x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "sns.set(font=\"lato\")\n",
    "sns.set_context(\"paper\")\n",
    "sns.set_style(\"whitegrid\")\n",
    "sns.set(font_scale=1.5)\n",
    "\n",
    "min_tn = (classes - 1.0) / float(classes)\n",
    "train_size = 0\n",
    "f = plt.figure(figsize=(7, 5))\n",
    "for idx, k in enumerate(models.keys()):\n",
    "    f = path.join(results_folder, \"{0}.npz\".format(k))\n",
    "    if not path.exists(f):\n",
    "        continue\n",
    "        \n",
    "    items = np.load(f)\n",
    "\n",
    "    ax = sns.lineplot(x=items['n'], y=items['ber'])\n",
    "\n",
    "    min_tn = min(min_tn, min(items['ber']))\n",
    "    train_size = max(train_size, max(items['n']))\n",
    "\n",
    "    ax.lines[-1].set_linestyle(\"--\")\n",
    "    ax.lines[-1].set_label(\"{}\".format(k))\n",
    "\n",
    "if target is not None:\n",
    "  x = [0, train_size]\n",
    "  sns.lineplot(x=x, y=[target]*len(x), ax=ax)\n",
    "  ax.lines[-1].set_color(\"Blue\")\n",
    "  ax.lines[-1].set_linestyle(\"--\")\n",
    "  ax.lines[-1].set_label(\"Target Error\")\n",
    "\n",
    "ax.legend()\n",
    "\n",
    "ax.set_ylim(0.0, min(1.0, max(target, min_tn)*2.0))\n",
    "ax.set_ylabel(\"BER Estimation\")\n",
    "ax.set_xlabel(\"Train Samples\")\n",
    "\n",
    "plt.setp(ax.get_legend().get_texts(), fontsize='12') # for legend text\n",
    "plt.setp(ax.get_legend().get_title(), fontsize='14') # for legend title"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87fa0f1c",
   "metadata": {},
   "source": [
    "## Explanations\n",
    "\n",
    "The plot above could indicate that it is very likely that even with many more samples, the curve using ```use``` as an embedding will still not converge below the target error and hence remains infeasible. If the best embedding did not converge though, one would want to gather more data to have a decisive answer. Alternatively, if a curve converged above a target accuracy (like in our case), the negative answer could still be off (i.e., false negative). This happens if the transformation bias increased the BER significantly (e.g., because it was trained on a too different soruce distributtion).\n",
    "False positive also can occure, although rarely seen in practice. The estimated bounds are lower bounds on the _true_ BER. It is known that these bounds are not tight always, therefore resulting in an overly optimistic answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a3cd42b",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}