rapids-plotly-paperspace-tutorial-3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "gradient": {}
   },
   "source": [
    "# RAPIDS + Plotly Dash on Paperspace ML-Driven Web Apps - Tutorial 3\n",
    "## Intro\n",
    "In Tutorial #2, we showed you how to kick off an ML workflow process that was accelerated by RAPIDS from your PLot.ly Dash webapp.  In this tutorial, we will exapnd on some of the available user interactions and visual outputs of Dash on the large dataset.\n",
    "\n",
    "## Install Extra Libraries\n",
    "The RAPIDSAI container is our starting point.  However, to build this dashboard, we need some extra libraries from plot.ly and jupyter.  You'll only need to install this once per container.  Leave uncommented if this is your first run.  Otherwise comment the `!pip install` lines to save time running. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install jupyter-dash tables plotly\n",
    "# !pip install dash dash-bootstrap-components dash-html-components matplotlib plotly\n",
    "# \n",
    "## To download data file\n",
    "# !wget https://rapidsai-data.s3.us-east-2.amazonaws.com/community-examples/plotly-webapp-demo/scrna_data.h5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "gradient": {}
   },
   "outputs": [],
   "source": [
    "import cudf\n",
    "import cuml\n",
    "import cupy as cp\n",
    "import numpy as np\n",
    "\n",
    "import plotly.graph_objects as go\n",
    "\n",
    "import dash\n",
    "import dash_bootstrap_components as dbc\n",
    "import dash_core_components as dcc\n",
    "import dash_html_components as html\n",
    "from dash.dependencies import Input, Output, State\n",
    "\n",
    "from flask import request\n",
    "\n",
    "from jupyter_dash import JupyterDash\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore', 'Expected ')\n",
    "warnings.simplefilter('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "gradient": {}
   },
   "source": [
    "In the previous two tutorials we learned:\n",
    " - Constructing an interactive application using Plotly and Dash components\n",
    " - Use RAPIDS for data preprocessing and ML\n",
    "\n",
    "In this tutorial we will inter twine UI and ML functions.\n",
    "\n",
    "# Import the Data\n",
    "We load Single Cell RNA data container cell wise(rows) genes(columns) counts. We will be using RAPIDS to infer information from the gene count matrix. Once the data is loaded, we arbitrarily select the first three genes as MARKER genes. We will use these MARKER genes data for comparision against processed data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "gradient": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.81 s, sys: 2.75 s, total: 7.57 s\n",
      "Wall time: 7.62 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Xkr4_val', 'Gm1992_val', 'Sox17_val']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "# If this cell fails, please download 'scrna_data.h5' from\n",
    "# https://rapidsai-data.s3.us-east-2.amazonaws.com/community-examples/plotly-webapp-demo/scrna_data.h5\n",
    "df = cudf.read_hdf('scrna_data.h5')\n",
    "df_markers = df.iloc[:, [0, 1, 2]]\n",
    "\n",
    "MARKERS = df.columns[0:3].tolist()\n",
    "MARKERS = [x + '_val' for x in MARKERS]\n",
    "\n",
    "df_markers.columns = MARKERS\n",
    "\n",
    "additional_cols = [x for x in MARKERS]\n",
    "additional_cols"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "gradient": {}
   },
   "source": [
    "# Process\n",
    "Now, we will use [cuML](https://docs.rapids.ai/api/cuml/stable/api.html) in RAPIDS to process the gene count matrix. First RAPIDS is used to perform PCA for dimensionality reduction and the reduced matrix is used to identify clusters using KMeans and learn the manifolds using UMAP. We also store the original index of the dataframe for correlation later to manage events in the dashboard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "gradient": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5.41 s, sys: 487 ms, total: 5.9 s\n",
      "Wall time: 5.9 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(100000, 57)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "n_pca=50\n",
    "n_clusters=30\n",
    "\n",
    "def cluster(df, n_pca_comp=50, n_clusters=30, additional_cols=None):\n",
    "\n",
    "    if 'orig_index' in df.columns:\n",
    "        print('Deleting orig_index')\n",
    "        idx = df['orig_index']\n",
    "        df.drop(['orig_index'], inplace=True, axis=1)\n",
    "    else:\n",
    "        idx = df.index\n",
    "    \n",
    "    prop_series, df = remove_non_feature_cols(df, additional_cols)\n",
    "    \n",
    "    pca = cuml.PCA(n_components=n_pca_comp)\n",
    "    pca.fit(df)\n",
    "    embedding = pca.transform(df)\n",
    "\n",
    "    kmeans_cuml = cuml.KMeans(n_clusters=n_clusters)\n",
    "    kmeans_cuml.fit(embedding)\n",
    "    kmeans_labels = kmeans_cuml.predict(embedding)\n",
    "\n",
    "    umap = cuml.manifold.UMAP()\n",
    "    Xt =  umap.fit_transform(embedding)\n",
    "\n",
    "    embedding['labels'] = kmeans_labels\n",
    "    embedding['x'] = Xt[0]\n",
    "    embedding['y'] = Xt[1]\n",
    "    embedding.index = idx\n",
    "    embedding['orig_index'] = idx\n",
    "    \n",
    "    for col in prop_series.keys():\n",
    "        embedding[col] = prop_series[col]\n",
    "    return embedding\n",
    "\n",
    "def remove_non_feature_cols(df, additional_cols):\n",
    "    prop_series = {}\n",
    "    for col_name in additional_cols:\n",
    "        if col_name in df.columns:\n",
    "            prop_series[col_name] = df[col_name]\n",
    "            df = df.drop([col_name], axis=1)\n",
    "\n",
    "    for col_name in ['labels', 'x', 'y']:\n",
    "        if col_name in df.columns:\n",
    "            df.drop([col_name], inplace=True, axis=1)\n",
    "            \n",
    "    return prop_series, df\n",
    "\n",
    "\n",
    "orig_df = cluster(df, n_pca_comp=n_pca, n_clusters=n_clusters, additional_cols=additional_cols)\n",
    "\n",
    "# Add marker genes back\n",
    "orig_df = cudf.concat([orig_df, df_markers], axis=1)\n",
    "orig_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "gradient": {}
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>47</th>\n",
       "      <th>48</th>\n",
       "      <th>49</th>\n",
       "      <th>labels</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>orig_index</th>\n",
       "      <th>Xkr4_val</th>\n",
       "      <th>Gm1992_val</th>\n",
       "      <th>Sox17_val</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.868425</td>\n",
       "      <td>0.336145</td>\n",
       "      <td>1.421227</td>\n",
       "      <td>-2.847102</td>\n",
       "      <td>-0.633875</td>\n",
       "      <td>3.891086</td>\n",
       "      <td>3.511035</td>\n",
       "      <td>-3.105289</td>\n",
       "      <td>-3.548421</td>\n",
       "      <td>0.086103</td>\n",
       "      <td>...</td>\n",
       "      <td>0.845912</td>\n",
       "      <td>1.195730</td>\n",
       "      <td>-1.716885</td>\n",
       "      <td>23</td>\n",
       "      <td>-2.693675</td>\n",
       "      <td>-0.198515</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.139578</td>\n",
       "      <td>-0.005757</td>\n",
       "      <td>0.049112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.660085</td>\n",
       "      <td>-0.411820</td>\n",
       "      <td>2.132230</td>\n",
       "      <td>-4.078625</td>\n",
       "      <td>-1.024646</td>\n",
       "      <td>3.650779</td>\n",
       "      <td>3.149760</td>\n",
       "      <td>-5.575364</td>\n",
       "      <td>0.577942</td>\n",
       "      <td>-0.278277</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.046324</td>\n",
       "      <td>-0.700149</td>\n",
       "      <td>-0.310466</td>\n",
       "      <td>9</td>\n",
       "      <td>-0.709105</td>\n",
       "      <td>0.085514</td>\n",
       "      <td>1</td>\n",
       "      <td>-0.130967</td>\n",
       "      <td>0.001012</td>\n",
       "      <td>0.045747</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13.293064</td>\n",
       "      <td>3.452299</td>\n",
       "      <td>-5.101409</td>\n",
       "      <td>-3.459800</td>\n",
       "      <td>0.248678</td>\n",
       "      <td>0.371149</td>\n",
       "      <td>1.881885</td>\n",
       "      <td>-2.333594</td>\n",
       "      <td>0.430030</td>\n",
       "      <td>2.331941</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.615356</td>\n",
       "      <td>-2.687400</td>\n",
       "      <td>-1.101635</td>\n",
       "      <td>12</td>\n",
       "      <td>-4.732382</td>\n",
       "      <td>5.286013</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.133182</td>\n",
       "      <td>0.000788</td>\n",
       "      <td>0.018643</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-4.962597</td>\n",
       "      <td>-1.094800</td>\n",
       "      <td>-1.184917</td>\n",
       "      <td>12.477499</td>\n",
       "      <td>0.852307</td>\n",
       "      <td>-2.062974</td>\n",
       "      <td>-0.705089</td>\n",
       "      <td>-7.453671</td>\n",
       "      <td>-0.326200</td>\n",
       "      <td>-1.523405</td>\n",
       "      <td>...</td>\n",
       "      <td>2.053468</td>\n",
       "      <td>-0.682838</td>\n",
       "      <td>1.046343</td>\n",
       "      <td>1</td>\n",
       "      <td>6.900117</td>\n",
       "      <td>4.495668</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.155867</td>\n",
       "      <td>-0.009243</td>\n",
       "      <td>-0.116312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-9.520083</td>\n",
       "      <td>0.326495</td>\n",
       "      <td>2.546679</td>\n",
       "      <td>-5.504425</td>\n",
       "      <td>-0.621280</td>\n",
       "      <td>0.145227</td>\n",
       "      <td>-0.697787</td>\n",
       "      <td>-4.823132</td>\n",
       "      <td>3.346006</td>\n",
       "      <td>0.645233</td>\n",
       "      <td>...</td>\n",
       "      <td>0.951622</td>\n",
       "      <td>-0.019756</td>\n",
       "      <td>-0.619416</td>\n",
       "      <td>3</td>\n",
       "      <td>2.419586</td>\n",
       "      <td>-2.063457</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.159265</td>\n",
       "      <td>-0.004439</td>\n",
       "      <td>-0.252790</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 57 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           0         1         2          3         4         5         6  \\\n",
       "0   1.868425  0.336145  1.421227  -2.847102 -0.633875  3.891086  3.511035   \n",
       "1   0.660085 -0.411820  2.132230  -4.078625 -1.024646  3.650779  3.149760   \n",
       "2  13.293064  3.452299 -5.101409  -3.459800  0.248678  0.371149  1.881885   \n",
       "3  -4.962597 -1.094800 -1.184917  12.477499  0.852307 -2.062974 -0.705089   \n",
       "4  -9.520083  0.326495  2.546679  -5.504425 -0.621280  0.145227 -0.697787   \n",
       "\n",
       "          7         8         9  ...        47        48        49  labels  \\\n",
       "0 -3.105289 -3.548421  0.086103  ...  0.845912  1.195730 -1.716885      23   \n",
       "1 -5.575364  0.577942 -0.278277  ... -0.046324 -0.700149 -0.310466       9   \n",
       "2 -2.333594  0.430030  2.331941  ... -0.615356 -2.687400 -1.101635      12   \n",
       "3 -7.453671 -0.326200 -1.523405  ...  2.053468 -0.682838  1.046343       1   \n",
       "4 -4.823132  3.346006  0.645233  ...  0.951622 -0.019756 -0.619416       3   \n",
       "\n",
       "          x         y  orig_index  Xkr4_val  Gm1992_val  Sox17_val  \n",
       "0 -2.693675 -0.198515           0 -0.139578   -0.005757   0.049112  \n",
       "1 -0.709105  0.085514           1 -0.130967    0.001012   0.045747  \n",
       "2 -4.732382  5.286013           2 -0.133182    0.000788   0.018643  \n",
       "3  6.900117  4.495668           3 -0.155867   -0.009243  -0.116312  \n",
       "4  2.419586 -2.063457           4 -0.159265   -0.004439  -0.252790  \n",
       "\n",
       "[5 rows x 57 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "orig_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "gradient": {}
   },
   "outputs": [],
   "source": [
    "# Utility function\n",
    "def generate_colors(num_colors):\n",
    "    \"\"\"\n",
    "    Generates evenly disributed colors\n",
    "    \"\"\"\n",
    "    a = ((np.random.random(size=num_colors) * 255))\n",
    "    b = ((np.random.random(size=num_colors) * 255))\n",
    "    return [\"#%02x%02x%02x\" % (int(r), int(g), 125) for r, g in zip(a, b)]\n",
    "\n",
    "colors = generate_colors(24)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's make scatter plot!\n",
    "\n",
    "The following method uses plotly to create a [scatter plot](https://plotly.com/python/line-and-scatter/). This plot will have the following properties:\n",
    "- Each cell is rendered as a point at the location(x and y) computed using UMAP\n",
    "- Color each cell using the 'label' generated using KMeans\n",
    "- Set cluster label as the text to be displayed on mouse over"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "gradient": {}
   },
   "outputs": [],
   "source": [
    "def generate_main_graph(df):\n",
    "    fig = go.Figure(layout = {'colorscale' : {}})\n",
    "    label_count = len(df['labels'].unique())\n",
    "    \n",
    "    for i in range(0, label_count):\n",
    "        si = str(i)\n",
    "        gdf = df.query('labels == ' + si)\n",
    "        fig.add_trace(\n",
    "            go.Scattergl({\n",
    "            'x': gdf['x'].to_array(),\n",
    "            'y': gdf['y'].to_array(),\n",
    "            'text': gdf['labels'].to_array(),\n",
    "            'customdata': gdf['orig_index'].to_array(),\n",
    "            'name': 'Cluster ' + si,\n",
    "            'mode': 'markers',\n",
    "            'marker': {'size': 2, 'color': colors[i % len(colors)]}\n",
    "        }))\n",
    "\n",
    "    fig.update_layout(\n",
    "            showlegend=True, clickmode='event', title='UMAP', dragmode='select',\n",
    "            annotations=[\n",
    "                dict(x=0.5, y=-0.07, showarrow=False, text='UMAP_1', xref=\"paper\", yref=\"paper\"),\n",
    "                dict(x=-0.05, y=0.5, showarrow=False, text=\"UMAP_2\", xref=\"paper\", yref=\"paper\", textangle=-90, )])\n",
    "    return fig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implement functions that will filter data based on user action and recompute UMAP and recluster the filtered data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "gradient": {}
   },
   "outputs": [],
   "source": [
    "def update_scatter_plots(df, additional_cols=None):\n",
    "    localdf = cluster(df, n_pca_comp=n_pca, n_clusters=n_clusters, additional_cols=additional_cols)\n",
    "\n",
    "    fig = generate_main_graph(localdf)\n",
    "    fig_marker1 = graph_scatter(localdf, MARKERS[0])\n",
    "    fig_marker2 = graph_scatter(localdf, MARKERS[1])\n",
    "    fig_marker3 = graph_scatter(localdf, MARKERS[2])\n",
    "\n",
    "    return localdf, fig, fig_marker1, fig_marker2, fig_marker3\n",
    "\n",
    "\n",
    "def update_scatter_plots_by_points(df, selected_point_indexes, additional_cols=None):\n",
    "    df = df[df['orig_index'].isin(selected_point_indexes)]\n",
    "    return update_scatter_plots(df, additional_cols=additional_cols)\n",
    "\n",
    "\n",
    "def update_scatter_plots_by_cluster_ids(df, selected_cluster_ids, additional_cols=None):\n",
    "    df = df[df['labels'].isin(selected_cluster_ids)]\n",
    "    return update_scatter_plots(df, additional_cols=additional_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create map for the marker genes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "gradient": {}
   },
   "outputs": [],
   "source": [
    "def graph_scatter(df, label):\n",
    "    \n",
    "    scatter_fig = go.Figure(layout = {'colorscale' : {}})\n",
    "    df[label] = df[label].round(3)\n",
    "    gdf = df.query(label + ' != 0').groupby(['labels',label], as_index=False).agg({'x':'count'})\n",
    "    \n",
    "    xmax = gdf['x'].max()\n",
    "    xmin = gdf['x'].min()\n",
    "    gdf = df.groupby(['labels', label], as_index=False).agg({'x':'count'})\n",
    "    \n",
    "    for i in range(0, len(gdf['labels'].unique())):\n",
    "        sdf = gdf.query('labels == ' + str(i))\n",
    "        _t = sdf['x'].astype('int')\n",
    "        _ace = (sdf[label] * 10).astype('int')\n",
    "        _z = cp.zeros(21)\n",
    "        cp.put(_z, _ace, _t)\n",
    "        _z[_z == 0] = None\n",
    "        scatter_fig.add_trace(\n",
    "                go.Scattergl({\n",
    "                'x': sdf[label].to_array(),\n",
    "                'y': sdf['labels'].to_array(),\n",
    "                'text': cp.asnumpy(_z),\n",
    "                'mode': 'markers',\n",
    "                'marker':{\n",
    "                    'size':4,\n",
    "                    'color':cp.asnumpy(_z),\n",
    "                    'colorscale':'Viridis',\n",
    "                    'cmin': xmin,\n",
    "                    'cmax': xmax,\n",
    "                    'showscale':True\n",
    "                }\n",
    "            }))\n",
    "\n",
    "    scatter_fig.update_layout(showlegend=False, clickmode='select', title=label, dragmode='select',\n",
    "        annotations=[\n",
    "            dict(x=0.5,   y=-0.15, showarrow=False, xref=\"paper\", yref=\"paper\", text=label),\n",
    "            dict(x=-0.15, y=0.5,   showarrow=False, xref=\"paper\", yref=\"paper\", text=\"Clusters\", textangle=-90)\n",
    "        ],)\n",
    "    return scatter_fig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "gradient": {}
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>47</th>\n",
       "      <th>48</th>\n",
       "      <th>49</th>\n",
       "      <th>labels</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>orig_index</th>\n",
       "      <th>Xkr4_val</th>\n",
       "      <th>Gm1992_val</th>\n",
       "      <th>Sox17_val</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.868425</td>\n",
       "      <td>0.336145</td>\n",
       "      <td>1.421227</td>\n",
       "      <td>-2.847102</td>\n",
       "      <td>-0.633875</td>\n",
       "      <td>3.891086</td>\n",
       "      <td>3.511035</td>\n",
       "      <td>-3.105289</td>\n",
       "      <td>-3.548421</td>\n",
       "      <td>0.086103</td>\n",
       "      <td>...</td>\n",
       "      <td>0.845912</td>\n",
       "      <td>1.195730</td>\n",
       "      <td>-1.716885</td>\n",
       "      <td>23</td>\n",
       "      <td>-2.693675</td>\n",
       "      <td>-0.198515</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.139578</td>\n",
       "      <td>-0.005757</td>\n",
       "      <td>0.049112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.660085</td>\n",
       "      <td>-0.411820</td>\n",
       "      <td>2.132230</td>\n",
       "      <td>-4.078625</td>\n",
       "      <td>-1.024646</td>\n",
       "      <td>3.650779</td>\n",
       "      <td>3.149760</td>\n",
       "      <td>-5.575364</td>\n",
       "      <td>0.577942</td>\n",
       "      <td>-0.278277</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.046324</td>\n",
       "      <td>-0.700149</td>\n",
       "      <td>-0.310466</td>\n",
       "      <td>9</td>\n",
       "      <td>-0.709105</td>\n",
       "      <td>0.085514</td>\n",
       "      <td>1</td>\n",
       "      <td>-0.130967</td>\n",
       "      <td>0.001012</td>\n",
       "      <td>0.045747</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13.293064</td>\n",
       "      <td>3.452299</td>\n",
       "      <td>-5.101409</td>\n",
       "      <td>-3.459800</td>\n",
       "      <td>0.248678</td>\n",
       "      <td>0.371149</td>\n",
       "      <td>1.881885</td>\n",
       "      <td>-2.333594</td>\n",
       "      <td>0.430030</td>\n",
       "      <td>2.331941</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.615356</td>\n",
       "      <td>-2.687400</td>\n",
       "      <td>-1.101635</td>\n",
       "      <td>12</td>\n",
       "      <td>-4.732382</td>\n",
       "      <td>5.286013</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.133182</td>\n",
       "      <td>0.000788</td>\n",
       "      <td>0.018643</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-4.962597</td>\n",
       "      <td>-1.094800</td>\n",
       "      <td>-1.184917</td>\n",
       "      <td>12.477499</td>\n",
       "      <td>0.852307</td>\n",
       "      <td>-2.062974</td>\n",
       "      <td>-0.705089</td>\n",
       "      <td>-7.453671</td>\n",
       "      <td>-0.326200</td>\n",
       "      <td>-1.523405</td>\n",
       "      <td>...</td>\n",
       "      <td>2.053468</td>\n",
       "      <td>-0.682838</td>\n",
       "      <td>1.046343</td>\n",
       "      <td>1</td>\n",
       "      <td>6.900117</td>\n",
       "      <td>4.495668</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.155867</td>\n",
       "      <td>-0.009243</td>\n",
       "      <td>-0.116312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-9.520083</td>\n",
       "      <td>0.326495</td>\n",
       "      <td>2.546679</td>\n",
       "      <td>-5.504425</td>\n",
       "      <td>-0.621280</td>\n",
       "      <td>0.145227</td>\n",
       "      <td>-0.697787</td>\n",
       "      <td>-4.823132</td>\n",
       "      <td>3.346006</td>\n",
       "      <td>0.645233</td>\n",
       "      <td>...</td>\n",
       "      <td>0.951622</td>\n",
       "      <td>-0.019756</td>\n",
       "      <td>-0.619416</td>\n",
       "      <td>3</td>\n",
       "      <td>2.419586</td>\n",
       "      <td>-2.063457</td>\n",
       "      <td>4</td>\n",
       "      <td>-0.159265</td>\n",
       "      <td>-0.004439</td>\n",
       "      <td>-0.252790</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 57 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           0         1         2          3         4         5         6  \\\n",
       "0   1.868425  0.336145  1.421227  -2.847102 -0.633875  3.891086  3.511035   \n",
       "1   0.660085 -0.411820  2.132230  -4.078625 -1.024646  3.650779  3.149760   \n",
       "2  13.293064  3.452299 -5.101409  -3.459800  0.248678  0.371149  1.881885   \n",
       "3  -4.962597 -1.094800 -1.184917  12.477499  0.852307 -2.062974 -0.705089   \n",
       "4  -9.520083  0.326495  2.546679  -5.504425 -0.621280  0.145227 -0.697787   \n",
       "\n",
       "          7         8         9  ...        47        48        49  labels  \\\n",
       "0 -3.105289 -3.548421  0.086103  ...  0.845912  1.195730 -1.716885      23   \n",
       "1 -5.575364  0.577942 -0.278277  ... -0.046324 -0.700149 -0.310466       9   \n",
       "2 -2.333594  0.430030  2.331941  ... -0.615356 -2.687400 -1.101635      12   \n",
       "3 -7.453671 -0.326200 -1.523405  ...  2.053468 -0.682838  1.046343       1   \n",
       "4 -4.823132  3.346006  0.645233  ...  0.951622 -0.019756 -0.619416       3   \n",
       "\n",
       "          x         y  orig_index  Xkr4_val  Gm1992_val  Sox17_val  \n",
       "0 -2.693675 -0.198515           0 -0.139578   -0.005757   0.049112  \n",
       "1 -0.709105  0.085514           1 -0.130967    0.001012   0.045747  \n",
       "2 -4.732382  5.286013           2 -0.133182    0.000788   0.018643  \n",
       "3  6.900117  4.495668           3 -0.155867   -0.009243  -0.116312  \n",
       "4  2.419586 -2.063457           4 -0.159265   -0.004439  -0.252790  \n",
       "\n",
       "[5 rows x 57 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tdf = orig_df.copy()\n",
    "tdf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's get that view on\n",
    "\n",
    "Please notice the 'recluster' callback method which integrates the UI events(data selection and commands) to trigger the recomputations using RAPIDs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "gradient": {}
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b style='color: red'>Please click on <a href='/proxy/8080/' target='_blank'>here</a> to open the dash</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dash app running on http://127.0.0.1:8080/proxy/8080/\n",
      "Event selectedData from basic-interactions\n",
      "Event - On selecting points on the main scatterplot\n",
      "Event selectedData from basic-interactions\n",
      "Event - On selecting points on the main scatterplot\n"
     ]
    }
   ],
   "source": [
    "# Please change the proxy_port if the port is already in use.\n",
    "proxy_port = 8080\n",
    "\n",
    "app = JupyterDash(__name__, \n",
    "                  external_stylesheets=['https://codepen.io/chriddyp/pen/bWLwgP.css', dbc.themes.BOOTSTRAP], \n",
    "                  requests_pathname_prefix='/proxy/' + str(proxy_port) + '/')\n",
    "# external_stylesheets = ['https://codepen.io/chriddyp/pen/bWLwgP.css', dbc.themes.BOOTSTRAP]\n",
    "# app = dash.Dash(__name__, external_stylesheets=external_stylesheets)\n",
    "\n",
    "fig = generate_main_graph(tdf)\n",
    "fig_ace = graph_scatter(tdf, MARKERS[0])\n",
    "fig_tmp = graph_scatter(tdf, MARKERS[1])\n",
    "fig_epc = graph_scatter(tdf, MARKERS[2])\n",
    "\n",
    "app.layout = html.Div(id='site_body', children=[\n",
    "    html.Div(className='row', children=[\n",
    "        html.Div([\n",
    "            dcc.Graph(id='basic-interactions', figure=fig),\n",
    "        ], className='nine columns', style={'verticalAlign': 'text-top',}),\n",
    "\n",
    "        html.Div([\n",
    "            html.Div(className='row', children=[\n",
    "                dbc.Button(\"Directions\", id=\"bt_open_directions\"),\n",
    "                dbc.Modal([\n",
    "                        dbc.ModalHeader(\"Directions\"),\n",
    "                        dbc.ModalBody(\n",
    "                            dcc.Markdown(\"\"\"\n",
    "                                The main scatterplot shows the UMAP visualization of single cells. \n",
    "                                \n",
    "                                ### Re-running Clustering and Visualization\n",
    "                                #### Reclusering by Clicking on Groups:\n",
    "                                1. Click on any point in a Cluster of Interest. The cluster to which that point belongs to will populate the Cluster field .  \n",
    "                                2. Click **Recluster on Selected Cluster**.\n",
    "                                #### Reclustering by Selecting Points: \n",
    "                                1. Use the **Box Select** or **Lasso Select** tool to select your points of interest. A number of points will populate the inthe Selected Points field .  \n",
    "                                2. Click **Recluster on Selected Points**.\n",
    "                                \n",
    "                                You can also manually add either Cluster number or selected points by typing it in. To do so, enter one or more numbers, e.g. “1” or “1,2,3” in the text box above the button for the operation you want to perform.\n",
    "                                \n",
    "                                ### Using the Toolbar\n",
    "                                Hover the mouse over the top right corner of the screen to see a toolbar. Hover over each tool to see its name. The tool options from left to right are:\n",
    "                                - **Camera:** download a snapshot of the current view as .png\n",
    "                                - **Zoom:** Click and drag to select a region of the plot to zoom into\n",
    "                                - **Pan:** Click and drag to shift the current view to a different region of the plot\n",
    "                                - **Box Select/Lasso Select:** both these tools can be used to select a region on the plot. The selected points are exported under ‘selection data’. See below to export the selected points to a dataframe.\n",
    "                                - **Zoom In/Zoom Out:** Zoom in and out centered on the current view.\n",
    "                            \"\"\"),\n",
    "                        ),\n",
    "                        dbc.ModalFooter(\n",
    "                            dbc.Button(\"Close\", id=\"bt_close_directions\", className=\"ml-auto\")\n",
    "                        ),\n",
    "                    ], id=\"md_directions\"),\n",
    "            ]),\n",
    "            \n",
    "            html.Div(className='row', children=[\n",
    "                dcc.Markdown(\"\"\"\n",
    "                    **Click Data**\n",
    "\n",
    "                    Click on points in the graph.\n",
    "                \"\"\"),\n",
    "            ], style={'marginTop': 18,}),\n",
    "            \n",
    "            html.Div(className='row', children=[\n",
    "                dcc.Input(id='cluster_id', type='text', style={'width': '80%',}),\n",
    "            ]),\n",
    "            \n",
    "            html.Div(className='row', children=[\n",
    "                dbc.Button('Recluster on Selected Clusters', id='recluster_by_id', n_clicks=0),\n",
    "            ], style={'marginTop': 6,}),\n",
    "            \n",
    "            html.Div(className='row', children=[\n",
    "                dcc.Markdown(\"\"\"\n",
    "                    **Selection Data**\n",
    "\n",
    "                    Choose the lasso or rectangle tool in the graph's menu\n",
    "                    bar and then select points in the graph.\n",
    "                \"\"\"),\n",
    "            ], style={'marginTop': 18,}),\n",
    "            html.Div(className='row', children=[\n",
    "                html.Div(dcc.Input(id='point_index', type='text')),\n",
    "            ]),\n",
    "            \n",
    "            html.Div(className='row', children=[html.Div(id='point_index_cnt'),]),\n",
    "            \n",
    "            html.Div(className='row', children=[\n",
    "                dbc.Button('Recluster on Selected Points', id='recluster_by_point', n_clicks=0),\n",
    "            ], style={'marginTop': 6,}),\n",
    "\n",
    "            html.Div(className='row', children=[dbc.Button(\"Exit\", id=\"bt_close\"),], style={'marginTop': 6,}),\n",
    "\n",
    "        ], className='three columns', style={'marginTop': 90, 'verticalAlign': 'text-top',}),\n",
    "    ]),\n",
    "\n",
    "    html.Div(className='row', children=[\n",
    "        html.Div([dcc.Graph(id='ace-interactions',figure=fig_ace)], className='four columns'),\n",
    "        html.Div([dcc.Graph(id='tmp-interactions',figure=fig_tmp)], className='four columns'),\n",
    "        html.Div([dcc.Graph(id='epc-interactions',figure=fig_epc)], className='four columns')\n",
    "    ]),\n",
    "])\n",
    "\n",
    "\n",
    "@app.callback(Output(\"site_body\", \"children\"), Input('bt_close', 'n_clicks'))\n",
    "def close_app(export_clicks):\n",
    "    if not dash.callback_context.triggered:\n",
    "        raise dash.exceptions.PreventUpdate\n",
    "\n",
    "    func = request.environ.get('werkzeug.server.shutdown')\n",
    "    if func is None:\n",
    "        raise RuntimeError('Not running with the Werkzeug Server')\n",
    "    func()\n",
    "    return 'Closed'\n",
    "\n",
    "\n",
    "@app.callback(\n",
    "    Output(\"md_directions\", \"is_open\"),\n",
    "    [Input(\"bt_open_directions\", \"n_clicks\"), Input(\"bt_close_directions\", \"n_clicks\")],\n",
    "    [State(\"md_directions\", \"is_open\")])\n",
    "def toggle_directions_dialog(n1, n2, is_open):\n",
    "    if n1 or n2:\n",
    "        return not is_open\n",
    "    return is_open\n",
    "\n",
    "\n",
    "@app.callback(\n",
    "    [Output('cluster_id', 'value'),\n",
    "     Output('point_index_cnt', 'children'),\n",
    "     Output('point_index', 'value')],\n",
    "    [Input('basic-interactions', 'clickData'),\n",
    "     Input('basic-interactions', 'selectedData')],\n",
    "    [State(\"cluster_id\", \"value\"), \n",
    "     State('point_index', 'value')])\n",
    "def display_selected_data(clicked_cluster, selected_point_index, \n",
    "                          selected_clusters, point_index):\n",
    "    if not dash.callback_context.triggered:\n",
    "        raise dash.exceptions.PreventUpdate\n",
    "\n",
    "    comp_id, event_type = dash.callback_context.triggered[0]['prop_id'].split('.')\n",
    "    print('Event %s from %s' % (event_type, comp_id))\n",
    "    \n",
    "    cluster_id = ''\n",
    "    point_cnt_str = ''\n",
    "    point_indexes = ''\n",
    "    \n",
    "    if comp_id == 'basic-interactions' and event_type == 'clickData':\n",
    "        # Event - On selecting cluster on the main scatter plot\n",
    "        if not selected_clusters:\n",
    "            selected_labels = []\n",
    "        else:\n",
    "            selected_labels = list(map(int, selected_clusters.split(\",\"))) \n",
    "        points = clicked_cluster['points']\n",
    "        for point in points:\n",
    "            selected_label = point['text']\n",
    "            if selected_label in selected_labels:\n",
    "                selected_labels.remove(selected_label)\n",
    "            else:\n",
    "                selected_labels.append(selected_label)\n",
    "        cluster_id = ','.join(map(str, selected_labels))\n",
    "        \n",
    "    elif comp_id == 'basic-interactions' and event_type == 'selectedData':\n",
    "        # Event - On selection on the main scatterplot\n",
    "        print('Event - On selecting points on the main scatterplot')\n",
    "        if not selected_point_index:\n",
    "            raise dash.exceptions.PreventUpdate\n",
    "\n",
    "        selected_point_indexes = []\n",
    "        for point in selected_point_index['points']:\n",
    "            selected_point_indexes.append(point['customdata'])\n",
    "            \n",
    "        if len(selected_point_indexes) <= 1:\n",
    "            raise dash.exceptions.PreventUpdate\n",
    "            \n",
    "        point_cnt_str = str(len(selected_point_indexes)) + ' points selected'\n",
    "        point_indexes = ', '.join(map(str, selected_point_indexes))\n",
    "     \n",
    "    else:\n",
    "        print('Unhandled event')\n",
    "        raise dash.exceptions.PreventUpdate\n",
    "        \n",
    "    return cluster_id, point_cnt_str, point_indexes #, fig_ace, fig_tmp, fig_epc\n",
    "\n",
    "\n",
    "@app.callback([Output('basic-interactions', 'figure'),\n",
    "               Output('ace-interactions', 'figure'),\n",
    "               Output('tmp-interactions', 'figure'),\n",
    "               Output('epc-interactions', 'figure')],\n",
    "              [Input('recluster_by_id', 'n_clicks'),\n",
    "               Input('recluster_by_point', 'n_clicks')],\n",
    "              [State(\"cluster_id\", \"value\"), \n",
    "               State('point_index', 'value')])\n",
    "def recluster(recluster_by_id, recluster_by_point, \n",
    "              cluster_id, point_index):\n",
    "    global tdf, fig_ace, fig_tmp, fig_epc\n",
    "\n",
    "    if not dash.callback_context.triggered:\n",
    "        raise dash.exceptions.PreventUpdate\n",
    "    comp_id, event_type = dash.callback_context.triggered[0]['prop_id'].split('.')\n",
    "    print('Event %s from %s' % (event_type, comp_id))\n",
    "\n",
    "    if comp_id == 'recluster_by_point' and event_type == 'n_clicks':\n",
    "        if not point_index:\n",
    "            raise dash.exceptions.PreventUpdate\n",
    "        # Event - On click 'recluster' button\n",
    "        selected_point_indexes = list(map(int, point_index.split(\",\")))\n",
    "        \n",
    "        tdf, fig, fig_ace, fig_tmp, fig_epc = update_scatter_plots_by_points(tdf, \n",
    "                                                                             selected_point_indexes, \n",
    "                                                                             additional_cols=additional_cols)\n",
    "    elif comp_id == 'recluster_by_id' and event_type == 'n_clicks':\n",
    "        if not cluster_id:\n",
    "            raise dash.exceptions.PreventUpdate\n",
    "        # Event - On click 'recluster' button\n",
    "        selected_cluster_ids = list(map(int, cluster_id.split(\",\")))\n",
    "        tdf, fig, fig_ace, fig_tmp, fig_epc = update_scatter_plots_by_cluster_ids(tdf, \n",
    "                                                                                  selected_cluster_ids,\n",
    "                                                                                  additional_cols=additional_cols)\n",
    "    else:\n",
    "        print('Unhandled event')\n",
    "        raise dash.exceptions.PreventUpdate\n",
    "    return fig, fig_ace, fig_tmp, fig_epc\n",
    "\n",
    "js = \"<b style='color: red'>Please click on <a href='/proxy/\" + str(proxy_port) + \"/' target='_blank'>here</a> to open the dash</b>\"\n",
    "display(HTML(js))\n",
    "\n",
    "srv = app.run_server(debug=True, use_reloader=False, port=proxy_port)\n",
    "srv"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}