non_nn_predictive.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "164d7c13",
   "metadata": {
    "id": "z5rr9teLTSjL"
   },
   "source": [
    "# Non Neural Network Predictive Model\n",
    "* We can now move on with our **predictive models**.\n",
    "* In this notebook we will try some regressors without using Neural Networks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81b666e3",
   "metadata": {
    "id": "Tj4ZML0N1FWh"
   },
   "source": [
    "## Import packages\n",
    "* To begin with, we will import the packages, that we will use in the following segments of the project:\n",
    "    * [pandas](https://pandas.pydata.org/)\n",
    "    * [NumPy](https://www.numpy.org/)\n",
    "    * [statsmodels](https://www.statsmodels.org/stable/index.html)\n",
    "    * [scikit-learn](https://scikit-learn.org/stable/)\n",
    "    * [XGBoost](https://xgboost.readthedocs.io/en/stable/)\n",
    "    * [LightGBM](https://lightgbm.readthedocs.io/en/latest/)\n",
    "* Note that the prementioned packages **must be locally installed too** in order to be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "16356b9f",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "L7Wja2PeyjKU",
    "outputId": "2636b116-d745-4253-e775-0d693a9064f7"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import statsmodels.api as sm\n",
    "import statsmodels.formula.api as smf\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "import sklearn.metrics\n",
    "from sklearn.cross_decomposition import PLSRegression\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.base import BaseEstimator, TransformerMixin\n",
    "import sklearn.metrics\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from sklearn.svm import LinearSVR\n",
    "from sklearn.model_selection import cross_validate\n",
    "\n",
    "import xgboost as xgb\n",
    "import lightgbm as lgb"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d89697a",
   "metadata": {},
   "source": [
    "## Data Load\n",
    "* We will use the files created in the <code>data_preparation</code> notebook.\n",
    "* This files can be found in the <code>data</code> folder.\n",
    "* Especially, we will load the following:\n",
    "  * <code>tracks.csv</code>\n",
    "  * <code>analyses_downsampled.csv</code>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "902730e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "tracks = pd.read_csv('data/tracks.csv', index_col=0)\n",
    "analyses_ds = pd.read_csv('data/analyses_downsampled.csv', index_col=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75a30eee",
   "metadata": {},
   "source": [
    "* We will merge these two files, in order to have the data into a single one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "bdbce27e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>danceability</th>\n",
       "      <th>energy</th>\n",
       "      <th>loudness</th>\n",
       "      <th>mode</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>acousticness</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>liveness</th>\n",
       "      <th>valence</th>\n",
       "      <th>tempo</th>\n",
       "      <th>...</th>\n",
       "      <th>timbre_7_std_mid</th>\n",
       "      <th>timbre_8_std_mid</th>\n",
       "      <th>timbre_9_std_mid</th>\n",
       "      <th>timbre_10_std_mid</th>\n",
       "      <th>timbre_11_std_mid</th>\n",
       "      <th>timbre_12_std_mid</th>\n",
       "      <th>loudness_start_std_mid</th>\n",
       "      <th>loudness_max_std_mid</th>\n",
       "      <th>loudness_max_time_std_mid</th>\n",
       "      <th>duration_std_mid</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0AJfVstyUONRJhyYgTkHMR</th>\n",
       "      <td>0.253</td>\n",
       "      <td>0.385</td>\n",
       "      <td>-8.031</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0341</td>\n",
       "      <td>0.729</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0428</td>\n",
       "      <td>0.6460</td>\n",
       "      <td>203.512</td>\n",
       "      <td>...</td>\n",
       "      <td>31.028081</td>\n",
       "      <td>21.500966</td>\n",
       "      <td>18.934411</td>\n",
       "      <td>19.095865</td>\n",
       "      <td>14.053563</td>\n",
       "      <td>16.839561</td>\n",
       "      <td>4.294521</td>\n",
       "      <td>3.070072</td>\n",
       "      <td>0.046489</td>\n",
       "      <td>0.104830</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1V3NH8aYfFyRjfNOHgoX3G</th>\n",
       "      <td>0.219</td>\n",
       "      <td>0.222</td>\n",
       "      <td>-16.941</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0390</td>\n",
       "      <td>0.978</td>\n",
       "      <td>0.000149</td>\n",
       "      <td>0.0837</td>\n",
       "      <td>0.0632</td>\n",
       "      <td>72.999</td>\n",
       "      <td>...</td>\n",
       "      <td>22.268559</td>\n",
       "      <td>11.564204</td>\n",
       "      <td>19.853243</td>\n",
       "      <td>9.968599</td>\n",
       "      <td>12.719616</td>\n",
       "      <td>23.051455</td>\n",
       "      <td>5.631639</td>\n",
       "      <td>5.568326</td>\n",
       "      <td>0.077225</td>\n",
       "      <td>0.259447</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4kfEy7h6bWBS5OaDpccTap</th>\n",
       "      <td>0.846</td>\n",
       "      <td>0.651</td>\n",
       "      <td>-5.481</td>\n",
       "      <td>0</td>\n",
       "      <td>0.1280</td>\n",
       "      <td>0.634</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.1250</td>\n",
       "      <td>0.5810</td>\n",
       "      <td>140.048</td>\n",
       "      <td>...</td>\n",
       "      <td>28.513058</td>\n",
       "      <td>29.713144</td>\n",
       "      <td>17.411532</td>\n",
       "      <td>23.796638</td>\n",
       "      <td>22.653034</td>\n",
       "      <td>18.470448</td>\n",
       "      <td>5.112536</td>\n",
       "      <td>2.455965</td>\n",
       "      <td>0.034787</td>\n",
       "      <td>0.084622</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7qgM0MdETmhd3mORTqvJRm</th>\n",
       "      <td>0.398</td>\n",
       "      <td>0.255</td>\n",
       "      <td>-11.205</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0373</td>\n",
       "      <td>0.944</td>\n",
       "      <td>0.000003</td>\n",
       "      <td>0.1140</td>\n",
       "      <td>0.3860</td>\n",
       "      <td>83.883</td>\n",
       "      <td>...</td>\n",
       "      <td>23.401036</td>\n",
       "      <td>22.397868</td>\n",
       "      <td>17.243494</td>\n",
       "      <td>16.434974</td>\n",
       "      <td>19.959498</td>\n",
       "      <td>13.918116</td>\n",
       "      <td>5.835958</td>\n",
       "      <td>5.094191</td>\n",
       "      <td>0.066420</td>\n",
       "      <td>0.140004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1HtgEaqki4dFclmVJlJDx1</th>\n",
       "      <td>0.612</td>\n",
       "      <td>0.579</td>\n",
       "      <td>-7.264</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0269</td>\n",
       "      <td>0.606</td>\n",
       "      <td>0.000009</td>\n",
       "      <td>0.3530</td>\n",
       "      <td>0.8100</td>\n",
       "      <td>136.641</td>\n",
       "      <td>...</td>\n",
       "      <td>24.076927</td>\n",
       "      <td>18.603327</td>\n",
       "      <td>15.858383</td>\n",
       "      <td>13.746648</td>\n",
       "      <td>16.996667</td>\n",
       "      <td>16.838443</td>\n",
       "      <td>4.014044</td>\n",
       "      <td>3.318366</td>\n",
       "      <td>0.037726</td>\n",
       "      <td>0.137322</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 136 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                        danceability  energy  loudness  mode  speechiness  \\\n",
       "id                                                                          \n",
       "0AJfVstyUONRJhyYgTkHMR         0.253   0.385    -8.031     0       0.0341   \n",
       "1V3NH8aYfFyRjfNOHgoX3G         0.219   0.222   -16.941     1       0.0390   \n",
       "4kfEy7h6bWBS5OaDpccTap         0.846   0.651    -5.481     0       0.1280   \n",
       "7qgM0MdETmhd3mORTqvJRm         0.398   0.255   -11.205     1       0.0373   \n",
       "1HtgEaqki4dFclmVJlJDx1         0.612   0.579    -7.264     1       0.0269   \n",
       "\n",
       "                        acousticness  instrumentalness  liveness  valence  \\\n",
       "id                                                                          \n",
       "0AJfVstyUONRJhyYgTkHMR         0.729          0.000000    0.0428   0.6460   \n",
       "1V3NH8aYfFyRjfNOHgoX3G         0.978          0.000149    0.0837   0.0632   \n",
       "4kfEy7h6bWBS5OaDpccTap         0.634          0.000000    0.1250   0.5810   \n",
       "7qgM0MdETmhd3mORTqvJRm         0.944          0.000003    0.1140   0.3860   \n",
       "1HtgEaqki4dFclmVJlJDx1         0.606          0.000009    0.3530   0.8100   \n",
       "\n",
       "                          tempo  ...  timbre_7_std_mid  timbre_8_std_mid  \\\n",
       "id                               ...                                       \n",
       "0AJfVstyUONRJhyYgTkHMR  203.512  ...         31.028081         21.500966   \n",
       "1V3NH8aYfFyRjfNOHgoX3G   72.999  ...         22.268559         11.564204   \n",
       "4kfEy7h6bWBS5OaDpccTap  140.048  ...         28.513058         29.713144   \n",
       "7qgM0MdETmhd3mORTqvJRm   83.883  ...         23.401036         22.397868   \n",
       "1HtgEaqki4dFclmVJlJDx1  136.641  ...         24.076927         18.603327   \n",
       "\n",
       "                        timbre_9_std_mid  timbre_10_std_mid  \\\n",
       "id                                                            \n",
       "0AJfVstyUONRJhyYgTkHMR         18.934411          19.095865   \n",
       "1V3NH8aYfFyRjfNOHgoX3G         19.853243           9.968599   \n",
       "4kfEy7h6bWBS5OaDpccTap         17.411532          23.796638   \n",
       "7qgM0MdETmhd3mORTqvJRm         17.243494          16.434974   \n",
       "1HtgEaqki4dFclmVJlJDx1         15.858383          13.746648   \n",
       "\n",
       "                        timbre_11_std_mid  timbre_12_std_mid  \\\n",
       "id                                                             \n",
       "0AJfVstyUONRJhyYgTkHMR          14.053563          16.839561   \n",
       "1V3NH8aYfFyRjfNOHgoX3G          12.719616          23.051455   \n",
       "4kfEy7h6bWBS5OaDpccTap          22.653034          18.470448   \n",
       "7qgM0MdETmhd3mORTqvJRm          19.959498          13.918116   \n",
       "1HtgEaqki4dFclmVJlJDx1          16.996667          16.838443   \n",
       "\n",
       "                        loudness_start_std_mid  loudness_max_std_mid  \\\n",
       "id                                                                     \n",
       "0AJfVstyUONRJhyYgTkHMR                4.294521              3.070072   \n",
       "1V3NH8aYfFyRjfNOHgoX3G                5.631639              5.568326   \n",
       "4kfEy7h6bWBS5OaDpccTap                5.112536              2.455965   \n",
       "7qgM0MdETmhd3mORTqvJRm                5.835958              5.094191   \n",
       "1HtgEaqki4dFclmVJlJDx1                4.014044              3.318366   \n",
       "\n",
       "                        loudness_max_time_std_mid  duration_std_mid  \n",
       "id                                                                   \n",
       "0AJfVstyUONRJhyYgTkHMR                   0.046489          0.104830  \n",
       "1V3NH8aYfFyRjfNOHgoX3G                   0.077225          0.259447  \n",
       "4kfEy7h6bWBS5OaDpccTap                   0.034787          0.084622  \n",
       "7qgM0MdETmhd3mORTqvJRm                   0.066420          0.140004  \n",
       "1HtgEaqki4dFclmVJlJDx1                   0.037726          0.137322  \n",
       "\n",
       "[5 rows x 136 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_tracks = tracks.join(analyses_ds)\n",
    "full_tracks.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4be1b069",
   "metadata": {},
   "source": [
    "## Dataset Generation\n",
    "* Here, it is necessary to define our dataset.\n",
    "* Specifically, we will use two Datasets, the first one will contain **all** the variables that we have, and for the second, we will choose the **important** ones.\n",
    "* We will lastly add another very simple model, that will contain **only the audio features of the songs**, and we will use it only as a base to evaluate the other models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39f83eeb",
   "metadata": {
    "id": "zYGpC4CnT_g3"
   },
   "source": [
    "### All Variables Dataset\n",
    "* First, we will create the dataset that contains all the variables that we have.\n",
    "* So we will use the <code>full_tracks</code> dataset.\n",
    "* We have to split our dataset into *train* and *test*.\n",
    "* The size of the test set will be the **20%** of the size of the whole dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "da9db5a6",
   "metadata": {
    "id": "OFm55mGmWhpv"
   },
   "outputs": [],
   "source": [
    "all_X, y = (full_tracks.drop(columns='valence'), full_tracks['valence'])\n",
    "\n",
    "all_X_train, all_X_test, y_train, y_test = train_test_split(all_X, y, test_size=0.2, \n",
    "                                                    random_state=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4bae514",
   "metadata": {
    "id": "o3mYBy49YeaL"
   },
   "source": [
    "### Important Variables Dataset\n",
    "* Then, we will create another dataset, that will contain only the variables that seem to have an impact on valence.\n",
    "* We will start by adding the audio features that affected valence the most, as we saw from the statistical analysis.\n",
    "* Specifically, we saw that **danceability**, **energy** and **acousticness** are important, so of course we will include them in the dataset.\n",
    "* Also, we will include **loudness**, **tempo** and **duration_ms** that have a less but still significant impact on *valence*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2eb719d6",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 237
    },
    "id": "TJ_8bnVtbAjA",
    "outputId": "6b62073a-c0d3-4168-a657-0bdb33d586a2"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>danceability</th>\n",
       "      <th>energy</th>\n",
       "      <th>acousticness</th>\n",
       "      <th>loudness</th>\n",
       "      <th>tempo</th>\n",
       "      <th>duration_ms</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0AJfVstyUONRJhyYgTkHMR</th>\n",
       "      <td>0.253</td>\n",
       "      <td>0.385</td>\n",
       "      <td>0.729</td>\n",
       "      <td>-8.031</td>\n",
       "      <td>203.512</td>\n",
       "      <td>161000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1V3NH8aYfFyRjfNOHgoX3G</th>\n",
       "      <td>0.219</td>\n",
       "      <td>0.222</td>\n",
       "      <td>0.978</td>\n",
       "      <td>-16.941</td>\n",
       "      <td>72.999</td>\n",
       "      <td>219707</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4kfEy7h6bWBS5OaDpccTap</th>\n",
       "      <td>0.846</td>\n",
       "      <td>0.651</td>\n",
       "      <td>0.634</td>\n",
       "      <td>-5.481</td>\n",
       "      <td>140.048</td>\n",
       "      <td>223921</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7qgM0MdETmhd3mORTqvJRm</th>\n",
       "      <td>0.398</td>\n",
       "      <td>0.255</td>\n",
       "      <td>0.944</td>\n",
       "      <td>-11.205</td>\n",
       "      <td>83.883</td>\n",
       "      <td>246573</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1HtgEaqki4dFclmVJlJDx1</th>\n",
       "      <td>0.612</td>\n",
       "      <td>0.579</td>\n",
       "      <td>0.606</td>\n",
       "      <td>-7.264</td>\n",
       "      <td>136.641</td>\n",
       "      <td>195531</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        danceability  energy  acousticness  loudness    tempo  \\\n",
       "id                                                                              \n",
       "0AJfVstyUONRJhyYgTkHMR         0.253   0.385         0.729    -8.031  203.512   \n",
       "1V3NH8aYfFyRjfNOHgoX3G         0.219   0.222         0.978   -16.941   72.999   \n",
       "4kfEy7h6bWBS5OaDpccTap         0.846   0.651         0.634    -5.481  140.048   \n",
       "7qgM0MdETmhd3mORTqvJRm         0.398   0.255         0.944   -11.205   83.883   \n",
       "1HtgEaqki4dFclmVJlJDx1         0.612   0.579         0.606    -7.264  136.641   \n",
       "\n",
       "                        duration_ms  \n",
       "id                                   \n",
       "0AJfVstyUONRJhyYgTkHMR       161000  \n",
       "1V3NH8aYfFyRjfNOHgoX3G       219707  \n",
       "4kfEy7h6bWBS5OaDpccTap       223921  \n",
       "7qgM0MdETmhd3mORTqvJRm       246573  \n",
       "1HtgEaqki4dFclmVJlJDx1       195531  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "important_audio_features = ['danceability', 'energy', 'acousticness', 'loudness', 'tempo', 'duration_ms']\n",
    "important_features_df = full_tracks[important_audio_features]\n",
    "important_features_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "829f9251",
   "metadata": {
    "id": "-vWP8Ia0br-q"
   },
   "source": [
    "* Then, we have to include also data from the deep analysis.\n",
    "* In order to have more **comprehensive** and also more **concentrated** information, we will use [PLSRegression](https://scikit-learn.org/dev/modules/generated/sklearn.cross_decomposition.PLSRegression.html#sklearn.cross_decomposition.PLSRegression). We will use this algorithm for dimensionality reduction. The reason, why we chose it here instead of PCA, is because it is a supervised algorithm and we want to capture in our dimensions, as much information as we can, to expain the variance of *valence*, and not generally the variance of our predictors.\n",
    "* But to do that correctly we first have to split our dataset into test and train, in order to train NCA only on the train data.\n",
    "* Here, we will use the split that we did previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c7630d22",
   "metadata": {
    "id": "38hlIcY5bjwZ"
   },
   "outputs": [],
   "source": [
    "train_ids = all_X_train.index\n",
    "test_ids = all_X_test.index\n",
    "train_analyses_df = analyses_ds.filter(items=train_ids, axis=0)\n",
    "test_analyses_df = analyses_ds.filter(items=test_ids, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d506f586",
   "metadata": {
    "id": "jF5noUOgoCGd"
   },
   "source": [
    "* Now we can execute our dimensionality reduction.\n",
    "* We will create 30 components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7ea05959",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 301
    },
    "id": "Yyipq5Qqn31x",
    "outputId": "e4a06d10-6d54-4d42-b34f-76040c93f9fd"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3c5i1PV78s4Mgz5DPvg5p8</th>\n",
       "      <td>0.218200</td>\n",
       "      <td>4.120789</td>\n",
       "      <td>2.410577</td>\n",
       "      <td>1.749678</td>\n",
       "      <td>-0.705838</td>\n",
       "      <td>-0.810013</td>\n",
       "      <td>-0.819584</td>\n",
       "      <td>1.090276</td>\n",
       "      <td>-0.550811</td>\n",
       "      <td>-0.620698</td>\n",
       "      <td>...</td>\n",
       "      <td>0.657423</td>\n",
       "      <td>0.899749</td>\n",
       "      <td>1.343165</td>\n",
       "      <td>-0.272157</td>\n",
       "      <td>0.926544</td>\n",
       "      <td>0.902467</td>\n",
       "      <td>0.833886</td>\n",
       "      <td>0.130493</td>\n",
       "      <td>-0.631656</td>\n",
       "      <td>1.065170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5W3Y5I9y5ps6M6rl09e7zg</th>\n",
       "      <td>-2.189106</td>\n",
       "      <td>1.951263</td>\n",
       "      <td>2.060270</td>\n",
       "      <td>-5.580001</td>\n",
       "      <td>-0.494353</td>\n",
       "      <td>-2.002322</td>\n",
       "      <td>-0.378216</td>\n",
       "      <td>-2.279076</td>\n",
       "      <td>-0.231867</td>\n",
       "      <td>1.033581</td>\n",
       "      <td>...</td>\n",
       "      <td>0.246576</td>\n",
       "      <td>-0.527923</td>\n",
       "      <td>-1.003374</td>\n",
       "      <td>1.470143</td>\n",
       "      <td>-0.455528</td>\n",
       "      <td>-0.961520</td>\n",
       "      <td>-0.121035</td>\n",
       "      <td>-0.818958</td>\n",
       "      <td>0.210470</td>\n",
       "      <td>0.181770</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1FssTKSXO3JLiQc3udfK5I</th>\n",
       "      <td>-6.102108</td>\n",
       "      <td>0.417266</td>\n",
       "      <td>1.352917</td>\n",
       "      <td>-1.263815</td>\n",
       "      <td>-1.273376</td>\n",
       "      <td>-1.772931</td>\n",
       "      <td>-0.769974</td>\n",
       "      <td>-1.044958</td>\n",
       "      <td>0.333710</td>\n",
       "      <td>2.249149</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.226232</td>\n",
       "      <td>0.699581</td>\n",
       "      <td>-0.254394</td>\n",
       "      <td>-0.419115</td>\n",
       "      <td>-0.136135</td>\n",
       "      <td>0.270881</td>\n",
       "      <td>-0.109923</td>\n",
       "      <td>-0.626718</td>\n",
       "      <td>0.428838</td>\n",
       "      <td>-0.043215</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0WWz2AaqxLoO0fa9ou6Fqc</th>\n",
       "      <td>0.507525</td>\n",
       "      <td>3.218161</td>\n",
       "      <td>4.134610</td>\n",
       "      <td>0.876526</td>\n",
       "      <td>-1.179215</td>\n",
       "      <td>0.180405</td>\n",
       "      <td>-0.398905</td>\n",
       "      <td>-0.772570</td>\n",
       "      <td>0.946695</td>\n",
       "      <td>-0.624516</td>\n",
       "      <td>...</td>\n",
       "      <td>0.280837</td>\n",
       "      <td>-0.014383</td>\n",
       "      <td>0.110718</td>\n",
       "      <td>0.283696</td>\n",
       "      <td>-0.457087</td>\n",
       "      <td>0.564088</td>\n",
       "      <td>1.585374</td>\n",
       "      <td>0.003861</td>\n",
       "      <td>0.833120</td>\n",
       "      <td>0.242135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5EeQ940KC0N4JCNLPIdRd1</th>\n",
       "      <td>-0.153389</td>\n",
       "      <td>0.908295</td>\n",
       "      <td>0.966328</td>\n",
       "      <td>1.847866</td>\n",
       "      <td>-4.404077</td>\n",
       "      <td>1.116358</td>\n",
       "      <td>0.391466</td>\n",
       "      <td>0.611845</td>\n",
       "      <td>0.195411</td>\n",
       "      <td>-0.911948</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.885772</td>\n",
       "      <td>-0.880681</td>\n",
       "      <td>0.072334</td>\n",
       "      <td>-0.121320</td>\n",
       "      <td>0.074630</td>\n",
       "      <td>-0.893758</td>\n",
       "      <td>-0.231688</td>\n",
       "      <td>0.229336</td>\n",
       "      <td>0.500625</td>\n",
       "      <td>0.129391</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 30 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                              0         1         2         3         4   \\\n",
       "id                                                                         \n",
       "3c5i1PV78s4Mgz5DPvg5p8  0.218200  4.120789  2.410577  1.749678 -0.705838   \n",
       "5W3Y5I9y5ps6M6rl09e7zg -2.189106  1.951263  2.060270 -5.580001 -0.494353   \n",
       "1FssTKSXO3JLiQc3udfK5I -6.102108  0.417266  1.352917 -1.263815 -1.273376   \n",
       "0WWz2AaqxLoO0fa9ou6Fqc  0.507525  3.218161  4.134610  0.876526 -1.179215   \n",
       "5EeQ940KC0N4JCNLPIdRd1 -0.153389  0.908295  0.966328  1.847866 -4.404077   \n",
       "\n",
       "                              5         6         7         8         9   ...  \\\n",
       "id                                                                        ...   \n",
       "3c5i1PV78s4Mgz5DPvg5p8 -0.810013 -0.819584  1.090276 -0.550811 -0.620698  ...   \n",
       "5W3Y5I9y5ps6M6rl09e7zg -2.002322 -0.378216 -2.279076 -0.231867  1.033581  ...   \n",
       "1FssTKSXO3JLiQc3udfK5I -1.772931 -0.769974 -1.044958  0.333710  2.249149  ...   \n",
       "0WWz2AaqxLoO0fa9ou6Fqc  0.180405 -0.398905 -0.772570  0.946695 -0.624516  ...   \n",
       "5EeQ940KC0N4JCNLPIdRd1  1.116358  0.391466  0.611845  0.195411 -0.911948  ...   \n",
       "\n",
       "                              20        21        22        23        24  \\\n",
       "id                                                                         \n",
       "3c5i1PV78s4Mgz5DPvg5p8  0.657423  0.899749  1.343165 -0.272157  0.926544   \n",
       "5W3Y5I9y5ps6M6rl09e7zg  0.246576 -0.527923 -1.003374  1.470143 -0.455528   \n",
       "1FssTKSXO3JLiQc3udfK5I -0.226232  0.699581 -0.254394 -0.419115 -0.136135   \n",
       "0WWz2AaqxLoO0fa9ou6Fqc  0.280837 -0.014383  0.110718  0.283696 -0.457087   \n",
       "5EeQ940KC0N4JCNLPIdRd1 -0.885772 -0.880681  0.072334 -0.121320  0.074630   \n",
       "\n",
       "                              25        26        27        28        29  \n",
       "id                                                                        \n",
       "3c5i1PV78s4Mgz5DPvg5p8  0.902467  0.833886  0.130493 -0.631656  1.065170  \n",
       "5W3Y5I9y5ps6M6rl09e7zg -0.961520 -0.121035 -0.818958  0.210470  0.181770  \n",
       "1FssTKSXO3JLiQc3udfK5I  0.270881 -0.109923 -0.626718  0.428838 -0.043215  \n",
       "0WWz2AaqxLoO0fa9ou6Fqc  0.564088  1.585374  0.003861  0.833120  0.242135  \n",
       "5EeQ940KC0N4JCNLPIdRd1 -0.893758 -0.231688  0.229336  0.500625  0.129391  \n",
       "\n",
       "[5 rows x 30 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plsr = PLSRegression(n_components=30)\n",
    "plsr.fit(train_analyses_df, y_train)\n",
    "train_analyses_reduced = pd.DataFrame(plsr.transform(train_analyses_df), index=train_ids)\n",
    "test_analyses_reduced = pd.DataFrame(plsr.transform(test_analyses_df), index=test_ids)\n",
    "\n",
    "train_analyses_reduced.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b090f71d",
   "metadata": {},
   "source": [
    "* We can see that the names of the columns are integers. A warning may be raised later because of that so we will transform them to strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "21de2079",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col_0</th>\n",
       "      <th>col_1</th>\n",
       "      <th>col_2</th>\n",
       "      <th>col_3</th>\n",
       "      <th>col_4</th>\n",
       "      <th>col_5</th>\n",
       "      <th>col_6</th>\n",
       "      <th>col_7</th>\n",
       "      <th>col_8</th>\n",
       "      <th>col_9</th>\n",
       "      <th>...</th>\n",
       "      <th>col_20</th>\n",
       "      <th>col_21</th>\n",
       "      <th>col_22</th>\n",
       "      <th>col_23</th>\n",
       "      <th>col_24</th>\n",
       "      <th>col_25</th>\n",
       "      <th>col_26</th>\n",
       "      <th>col_27</th>\n",
       "      <th>col_28</th>\n",
       "      <th>col_29</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3c5i1PV78s4Mgz5DPvg5p8</th>\n",
       "      <td>0.218200</td>\n",
       "      <td>4.120789</td>\n",
       "      <td>2.410577</td>\n",
       "      <td>1.749678</td>\n",
       "      <td>-0.705838</td>\n",
       "      <td>-0.810013</td>\n",
       "      <td>-0.819584</td>\n",
       "      <td>1.090276</td>\n",
       "      <td>-0.550811</td>\n",
       "      <td>-0.620698</td>\n",
       "      <td>...</td>\n",
       "      <td>0.657423</td>\n",
       "      <td>0.899749</td>\n",
       "      <td>1.343165</td>\n",
       "      <td>-0.272157</td>\n",
       "      <td>0.926544</td>\n",
       "      <td>0.902467</td>\n",
       "      <td>0.833886</td>\n",
       "      <td>0.130493</td>\n",
       "      <td>-0.631656</td>\n",
       "      <td>1.065170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5W3Y5I9y5ps6M6rl09e7zg</th>\n",
       "      <td>-2.189106</td>\n",
       "      <td>1.951263</td>\n",
       "      <td>2.060270</td>\n",
       "      <td>-5.580001</td>\n",
       "      <td>-0.494353</td>\n",
       "      <td>-2.002322</td>\n",
       "      <td>-0.378216</td>\n",
       "      <td>-2.279076</td>\n",
       "      <td>-0.231867</td>\n",
       "      <td>1.033581</td>\n",
       "      <td>...</td>\n",
       "      <td>0.246576</td>\n",
       "      <td>-0.527923</td>\n",
       "      <td>-1.003374</td>\n",
       "      <td>1.470143</td>\n",
       "      <td>-0.455528</td>\n",
       "      <td>-0.961520</td>\n",
       "      <td>-0.121035</td>\n",
       "      <td>-0.818958</td>\n",
       "      <td>0.210470</td>\n",
       "      <td>0.181770</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1FssTKSXO3JLiQc3udfK5I</th>\n",
       "      <td>-6.102108</td>\n",
       "      <td>0.417266</td>\n",
       "      <td>1.352917</td>\n",
       "      <td>-1.263815</td>\n",
       "      <td>-1.273376</td>\n",
       "      <td>-1.772931</td>\n",
       "      <td>-0.769974</td>\n",
       "      <td>-1.044958</td>\n",
       "      <td>0.333710</td>\n",
       "      <td>2.249149</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.226232</td>\n",
       "      <td>0.699581</td>\n",
       "      <td>-0.254394</td>\n",
       "      <td>-0.419115</td>\n",
       "      <td>-0.136135</td>\n",
       "      <td>0.270881</td>\n",
       "      <td>-0.109923</td>\n",
       "      <td>-0.626718</td>\n",
       "      <td>0.428838</td>\n",
       "      <td>-0.043215</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0WWz2AaqxLoO0fa9ou6Fqc</th>\n",
       "      <td>0.507525</td>\n",
       "      <td>3.218161</td>\n",
       "      <td>4.134610</td>\n",
       "      <td>0.876526</td>\n",
       "      <td>-1.179215</td>\n",
       "      <td>0.180405</td>\n",
       "      <td>-0.398905</td>\n",
       "      <td>-0.772570</td>\n",
       "      <td>0.946695</td>\n",
       "      <td>-0.624516</td>\n",
       "      <td>...</td>\n",
       "      <td>0.280837</td>\n",
       "      <td>-0.014383</td>\n",
       "      <td>0.110718</td>\n",
       "      <td>0.283696</td>\n",
       "      <td>-0.457087</td>\n",
       "      <td>0.564088</td>\n",
       "      <td>1.585374</td>\n",
       "      <td>0.003861</td>\n",
       "      <td>0.833120</td>\n",
       "      <td>0.242135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5EeQ940KC0N4JCNLPIdRd1</th>\n",
       "      <td>-0.153389</td>\n",
       "      <td>0.908295</td>\n",
       "      <td>0.966328</td>\n",
       "      <td>1.847866</td>\n",
       "      <td>-4.404077</td>\n",
       "      <td>1.116358</td>\n",
       "      <td>0.391466</td>\n",
       "      <td>0.611845</td>\n",
       "      <td>0.195411</td>\n",
       "      <td>-0.911948</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.885772</td>\n",
       "      <td>-0.880681</td>\n",
       "      <td>0.072334</td>\n",
       "      <td>-0.121320</td>\n",
       "      <td>0.074630</td>\n",
       "      <td>-0.893758</td>\n",
       "      <td>-0.231688</td>\n",
       "      <td>0.229336</td>\n",
       "      <td>0.500625</td>\n",
       "      <td>0.129391</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 30 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                           col_0     col_1     col_2     col_3     col_4  \\\n",
       "id                                                                         \n",
       "3c5i1PV78s4Mgz5DPvg5p8  0.218200  4.120789  2.410577  1.749678 -0.705838   \n",
       "5W3Y5I9y5ps6M6rl09e7zg -2.189106  1.951263  2.060270 -5.580001 -0.494353   \n",
       "1FssTKSXO3JLiQc3udfK5I -6.102108  0.417266  1.352917 -1.263815 -1.273376   \n",
       "0WWz2AaqxLoO0fa9ou6Fqc  0.507525  3.218161  4.134610  0.876526 -1.179215   \n",
       "5EeQ940KC0N4JCNLPIdRd1 -0.153389  0.908295  0.966328  1.847866 -4.404077   \n",
       "\n",
       "                           col_5     col_6     col_7     col_8     col_9  ...  \\\n",
       "id                                                                        ...   \n",
       "3c5i1PV78s4Mgz5DPvg5p8 -0.810013 -0.819584  1.090276 -0.550811 -0.620698  ...   \n",
       "5W3Y5I9y5ps6M6rl09e7zg -2.002322 -0.378216 -2.279076 -0.231867  1.033581  ...   \n",
       "1FssTKSXO3JLiQc3udfK5I -1.772931 -0.769974 -1.044958  0.333710  2.249149  ...   \n",
       "0WWz2AaqxLoO0fa9ou6Fqc  0.180405 -0.398905 -0.772570  0.946695 -0.624516  ...   \n",
       "5EeQ940KC0N4JCNLPIdRd1  1.116358  0.391466  0.611845  0.195411 -0.911948  ...   \n",
       "\n",
       "                          col_20    col_21    col_22    col_23    col_24  \\\n",
       "id                                                                         \n",
       "3c5i1PV78s4Mgz5DPvg5p8  0.657423  0.899749  1.343165 -0.272157  0.926544   \n",
       "5W3Y5I9y5ps6M6rl09e7zg  0.246576 -0.527923 -1.003374  1.470143 -0.455528   \n",
       "1FssTKSXO3JLiQc3udfK5I -0.226232  0.699581 -0.254394 -0.419115 -0.136135   \n",
       "0WWz2AaqxLoO0fa9ou6Fqc  0.280837 -0.014383  0.110718  0.283696 -0.457087   \n",
       "5EeQ940KC0N4JCNLPIdRd1 -0.885772 -0.880681  0.072334 -0.121320  0.074630   \n",
       "\n",
       "                          col_25    col_26    col_27    col_28    col_29  \n",
       "id                                                                        \n",
       "3c5i1PV78s4Mgz5DPvg5p8  0.902467  0.833886  0.130493 -0.631656  1.065170  \n",
       "5W3Y5I9y5ps6M6rl09e7zg -0.961520 -0.121035 -0.818958  0.210470  0.181770  \n",
       "1FssTKSXO3JLiQc3udfK5I  0.270881 -0.109923 -0.626718  0.428838 -0.043215  \n",
       "0WWz2AaqxLoO0fa9ou6Fqc  0.564088  1.585374  0.003861  0.833120  0.242135  \n",
       "5EeQ940KC0N4JCNLPIdRd1 -0.893758 -0.231688  0.229336  0.500625  0.129391  \n",
       "\n",
       "[5 rows x 30 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def make_columns_strings(df):\n",
    "    df.columns = map(lambda x: 'col_' + str(x), list(df.columns))\n",
    "\n",
    "make_columns_strings(train_analyses_reduced)\n",
    "make_columns_strings(test_analyses_reduced)\n",
    "\n",
    "train_analyses_reduced.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5b75594",
   "metadata": {
    "id": "iSQrBtfIo0Ox"
   },
   "source": [
    "* Following, we will merge our important audio features with the reduced dimensions of deep analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b0f275f6",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 301
    },
    "id": "GNtvT26sonkl",
    "outputId": "fb393e15-7a98-4a43-e32d-0a81ce1965e7"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col_0</th>\n",
       "      <th>col_1</th>\n",
       "      <th>col_2</th>\n",
       "      <th>col_3</th>\n",
       "      <th>col_4</th>\n",
       "      <th>col_5</th>\n",
       "      <th>col_6</th>\n",
       "      <th>col_7</th>\n",
       "      <th>col_8</th>\n",
       "      <th>col_9</th>\n",
       "      <th>...</th>\n",
       "      <th>col_26</th>\n",
       "      <th>col_27</th>\n",
       "      <th>col_28</th>\n",
       "      <th>col_29</th>\n",
       "      <th>danceability</th>\n",
       "      <th>energy</th>\n",
       "      <th>acousticness</th>\n",
       "      <th>loudness</th>\n",
       "      <th>tempo</th>\n",
       "      <th>duration_ms</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3c5i1PV78s4Mgz5DPvg5p8</th>\n",
       "      <td>0.218200</td>\n",
       "      <td>4.120789</td>\n",
       "      <td>2.410577</td>\n",
       "      <td>1.749678</td>\n",
       "      <td>-0.705838</td>\n",
       "      <td>-0.810013</td>\n",
       "      <td>-0.819584</td>\n",
       "      <td>1.090276</td>\n",
       "      <td>-0.550811</td>\n",
       "      <td>-0.620698</td>\n",
       "      <td>...</td>\n",
       "      <td>0.833886</td>\n",
       "      <td>0.130493</td>\n",
       "      <td>-0.631656</td>\n",
       "      <td>1.065170</td>\n",
       "      <td>0.218</td>\n",
       "      <td>0.939</td>\n",
       "      <td>0.000134</td>\n",
       "      <td>-4.524</td>\n",
       "      <td>174.393</td>\n",
       "      <td>243320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5W3Y5I9y5ps6M6rl09e7zg</th>\n",
       "      <td>-2.189106</td>\n",
       "      <td>1.951263</td>\n",
       "      <td>2.060270</td>\n",
       "      <td>-5.580001</td>\n",
       "      <td>-0.494353</td>\n",
       "      <td>-2.002322</td>\n",
       "      <td>-0.378216</td>\n",
       "      <td>-2.279076</td>\n",
       "      <td>-0.231867</td>\n",
       "      <td>1.033581</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.121035</td>\n",
       "      <td>-0.818958</td>\n",
       "      <td>0.210470</td>\n",
       "      <td>0.181770</td>\n",
       "      <td>0.865</td>\n",
       "      <td>0.394</td>\n",
       "      <td>0.083200</td>\n",
       "      <td>-12.235</td>\n",
       "      <td>115.936</td>\n",
       "      <td>521693</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1FssTKSXO3JLiQc3udfK5I</th>\n",
       "      <td>-6.102108</td>\n",
       "      <td>0.417266</td>\n",
       "      <td>1.352917</td>\n",
       "      <td>-1.263815</td>\n",
       "      <td>-1.273376</td>\n",
       "      <td>-1.772931</td>\n",
       "      <td>-0.769974</td>\n",
       "      <td>-1.044958</td>\n",
       "      <td>0.333710</td>\n",
       "      <td>2.249149</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.109923</td>\n",
       "      <td>-0.626718</td>\n",
       "      <td>0.428838</td>\n",
       "      <td>-0.043215</td>\n",
       "      <td>0.834</td>\n",
       "      <td>0.775</td>\n",
       "      <td>0.006940</td>\n",
       "      <td>-6.144</td>\n",
       "      <td>133.979</td>\n",
       "      <td>220227</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0WWz2AaqxLoO0fa9ou6Fqc</th>\n",
       "      <td>0.507525</td>\n",
       "      <td>3.218161</td>\n",
       "      <td>4.134610</td>\n",
       "      <td>0.876526</td>\n",
       "      <td>-1.179215</td>\n",
       "      <td>0.180405</td>\n",
       "      <td>-0.398905</td>\n",
       "      <td>-0.772570</td>\n",
       "      <td>0.946695</td>\n",
       "      <td>-0.624516</td>\n",
       "      <td>...</td>\n",
       "      <td>1.585374</td>\n",
       "      <td>0.003861</td>\n",
       "      <td>0.833120</td>\n",
       "      <td>0.242135</td>\n",
       "      <td>0.491</td>\n",
       "      <td>0.783</td>\n",
       "      <td>0.005210</td>\n",
       "      <td>-8.997</td>\n",
       "      <td>127.437</td>\n",
       "      <td>328220</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5EeQ940KC0N4JCNLPIdRd1</th>\n",
       "      <td>-0.153389</td>\n",
       "      <td>0.908295</td>\n",
       "      <td>0.966328</td>\n",
       "      <td>1.847866</td>\n",
       "      <td>-4.404077</td>\n",
       "      <td>1.116358</td>\n",
       "      <td>0.391466</td>\n",
       "      <td>0.611845</td>\n",
       "      <td>0.195411</td>\n",
       "      <td>-0.911948</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.231688</td>\n",
       "      <td>0.229336</td>\n",
       "      <td>0.500625</td>\n",
       "      <td>0.129391</td>\n",
       "      <td>0.379</td>\n",
       "      <td>0.901</td>\n",
       "      <td>0.015700</td>\n",
       "      <td>-5.777</td>\n",
       "      <td>169.970</td>\n",
       "      <td>261960</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                           col_0     col_1     col_2     col_3     col_4  \\\n",
       "id                                                                         \n",
       "3c5i1PV78s4Mgz5DPvg5p8  0.218200  4.120789  2.410577  1.749678 -0.705838   \n",
       "5W3Y5I9y5ps6M6rl09e7zg -2.189106  1.951263  2.060270 -5.580001 -0.494353   \n",
       "1FssTKSXO3JLiQc3udfK5I -6.102108  0.417266  1.352917 -1.263815 -1.273376   \n",
       "0WWz2AaqxLoO0fa9ou6Fqc  0.507525  3.218161  4.134610  0.876526 -1.179215   \n",
       "5EeQ940KC0N4JCNLPIdRd1 -0.153389  0.908295  0.966328  1.847866 -4.404077   \n",
       "\n",
       "                           col_5     col_6     col_7     col_8     col_9  ...  \\\n",
       "id                                                                        ...   \n",
       "3c5i1PV78s4Mgz5DPvg5p8 -0.810013 -0.819584  1.090276 -0.550811 -0.620698  ...   \n",
       "5W3Y5I9y5ps6M6rl09e7zg -2.002322 -0.378216 -2.279076 -0.231867  1.033581  ...   \n",
       "1FssTKSXO3JLiQc3udfK5I -1.772931 -0.769974 -1.044958  0.333710  2.249149  ...   \n",
       "0WWz2AaqxLoO0fa9ou6Fqc  0.180405 -0.398905 -0.772570  0.946695 -0.624516  ...   \n",
       "5EeQ940KC0N4JCNLPIdRd1  1.116358  0.391466  0.611845  0.195411 -0.911948  ...   \n",
       "\n",
       "                          col_26    col_27    col_28    col_29  danceability  \\\n",
       "id                                                                             \n",
       "3c5i1PV78s4Mgz5DPvg5p8  0.833886  0.130493 -0.631656  1.065170         0.218   \n",
       "5W3Y5I9y5ps6M6rl09e7zg -0.121035 -0.818958  0.210470  0.181770         0.865   \n",
       "1FssTKSXO3JLiQc3udfK5I -0.109923 -0.626718  0.428838 -0.043215         0.834   \n",
       "0WWz2AaqxLoO0fa9ou6Fqc  1.585374  0.003861  0.833120  0.242135         0.491   \n",
       "5EeQ940KC0N4JCNLPIdRd1 -0.231688  0.229336  0.500625  0.129391         0.379   \n",
       "\n",
       "                        energy  acousticness  loudness    tempo  duration_ms  \n",
       "id                                                                            \n",
       "3c5i1PV78s4Mgz5DPvg5p8   0.939      0.000134    -4.524  174.393       243320  \n",
       "5W3Y5I9y5ps6M6rl09e7zg   0.394      0.083200   -12.235  115.936       521693  \n",
       "1FssTKSXO3JLiQc3udfK5I   0.775      0.006940    -6.144  133.979       220227  \n",
       "0WWz2AaqxLoO0fa9ou6Fqc   0.783      0.005210    -8.997  127.437       328220  \n",
       "5EeQ940KC0N4JCNLPIdRd1   0.901      0.015700    -5.777  169.970       261960  \n",
       "\n",
       "[5 rows x 36 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imp_X_train = train_analyses_reduced.join(important_features_df)\n",
    "imp_X_test = test_analyses_reduced.join(important_features_df)\n",
    "imp_X_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c553a10d",
   "metadata": {
    "id": "u_fDP3VQMrOz"
   },
   "source": [
    "### Simple Audio Features Model\n",
    "* As we mentioned before we will use another model that contains only the audio features, to evaluate our results.\n",
    "* *Reminder: Before starting our statistical analysis we created the <code>independent</code> collection, that contains the names of the audio features, without valence.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "448bc21a",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "qY72x6BaNJe6",
    "outputId": "cb767492-c02b-4b40-cfee-16ebb62bdbf6"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['danceability', 'energy', 'loudness', 'mode', 'speechiness',\n",
       "       'acousticness', 'instrumentalness', 'liveness', 'tempo', 'duration_ms',\n",
       "       'time_signature', 'key_0', 'key_1', 'key_2', 'key_3', 'key_4', 'key_5',\n",
       "       'key_6', 'key_7', 'key_8', 'key_9', 'key_10', 'key_11'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "audio_features = tracks.columns.drop('valence')\n",
    "audio_features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "084718de",
   "metadata": {
    "id": "koClAMhENnV5"
   },
   "source": [
    "* In our simple model, we will include these columns only.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6ec0bc49",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 301
    },
    "id": "OyYH8-W7NscO",
    "outputId": "782b38ea-f8ce-4552-ecc1-14efcaa52f56"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>danceability</th>\n",
       "      <th>energy</th>\n",
       "      <th>loudness</th>\n",
       "      <th>mode</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>acousticness</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>liveness</th>\n",
       "      <th>tempo</th>\n",
       "      <th>duration_ms</th>\n",
       "      <th>...</th>\n",
       "      <th>key_2</th>\n",
       "      <th>key_3</th>\n",
       "      <th>key_4</th>\n",
       "      <th>key_5</th>\n",
       "      <th>key_6</th>\n",
       "      <th>key_7</th>\n",
       "      <th>key_8</th>\n",
       "      <th>key_9</th>\n",
       "      <th>key_10</th>\n",
       "      <th>key_11</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3c5i1PV78s4Mgz5DPvg5p8</th>\n",
       "      <td>0.218</td>\n",
       "      <td>0.939</td>\n",
       "      <td>-4.524</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0866</td>\n",
       "      <td>0.000134</td>\n",
       "      <td>0.002290</td>\n",
       "      <td>0.1100</td>\n",
       "      <td>174.393</td>\n",
       "      <td>243320</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5W3Y5I9y5ps6M6rl09e7zg</th>\n",
       "      <td>0.865</td>\n",
       "      <td>0.394</td>\n",
       "      <td>-12.235</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0528</td>\n",
       "      <td>0.083200</td>\n",
       "      <td>0.002310</td>\n",
       "      <td>0.3020</td>\n",
       "      <td>115.936</td>\n",
       "      <td>521693</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1FssTKSXO3JLiQc3udfK5I</th>\n",
       "      <td>0.834</td>\n",
       "      <td>0.775</td>\n",
       "      <td>-6.144</td>\n",
       "      <td>0</td>\n",
       "      <td>0.1800</td>\n",
       "      <td>0.006940</td>\n",
       "      <td>0.000024</td>\n",
       "      <td>0.4760</td>\n",
       "      <td>133.979</td>\n",
       "      <td>220227</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0WWz2AaqxLoO0fa9ou6Fqc</th>\n",
       "      <td>0.491</td>\n",
       "      <td>0.783</td>\n",
       "      <td>-8.997</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0484</td>\n",
       "      <td>0.005210</td>\n",
       "      <td>0.036100</td>\n",
       "      <td>0.0951</td>\n",
       "      <td>127.437</td>\n",
       "      <td>328220</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5EeQ940KC0N4JCNLPIdRd1</th>\n",
       "      <td>0.379</td>\n",
       "      <td>0.901</td>\n",
       "      <td>-5.777</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0522</td>\n",
       "      <td>0.015700</td>\n",
       "      <td>0.000084</td>\n",
       "      <td>0.0779</td>\n",
       "      <td>169.970</td>\n",
       "      <td>261960</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                        danceability  energy  loudness  mode  speechiness  \\\n",
       "id                                                                          \n",
       "3c5i1PV78s4Mgz5DPvg5p8         0.218   0.939    -4.524     0       0.0866   \n",
       "5W3Y5I9y5ps6M6rl09e7zg         0.865   0.394   -12.235     1       0.0528   \n",
       "1FssTKSXO3JLiQc3udfK5I         0.834   0.775    -6.144     0       0.1800   \n",
       "0WWz2AaqxLoO0fa9ou6Fqc         0.491   0.783    -8.997     0       0.0484   \n",
       "5EeQ940KC0N4JCNLPIdRd1         0.379   0.901    -5.777     1       0.0522   \n",
       "\n",
       "                        acousticness  instrumentalness  liveness    tempo  \\\n",
       "id                                                                          \n",
       "3c5i1PV78s4Mgz5DPvg5p8      0.000134          0.002290    0.1100  174.393   \n",
       "5W3Y5I9y5ps6M6rl09e7zg      0.083200          0.002310    0.3020  115.936   \n",
       "1FssTKSXO3JLiQc3udfK5I      0.006940          0.000024    0.4760  133.979   \n",
       "0WWz2AaqxLoO0fa9ou6Fqc      0.005210          0.036100    0.0951  127.437   \n",
       "5EeQ940KC0N4JCNLPIdRd1      0.015700          0.000084    0.0779  169.970   \n",
       "\n",
       "                        duration_ms  ...  key_2  key_3  key_4  key_5  key_6  \\\n",
       "id                                   ...                                      \n",
       "3c5i1PV78s4Mgz5DPvg5p8       243320  ...      0      0      0      0      0   \n",
       "5W3Y5I9y5ps6M6rl09e7zg       521693  ...      0      0      0      0      0   \n",
       "1FssTKSXO3JLiQc3udfK5I       220227  ...      0      0      0      0      0   \n",
       "0WWz2AaqxLoO0fa9ou6Fqc       328220  ...      0      0      0      0      0   \n",
       "5EeQ940KC0N4JCNLPIdRd1       261960  ...      0      0      1      0      0   \n",
       "\n",
       "                        key_7  key_8  key_9  key_10  key_11  \n",
       "id                                                           \n",
       "3c5i1PV78s4Mgz5DPvg5p8      0      0      0       1       0  \n",
       "5W3Y5I9y5ps6M6rl09e7zg      0      0      0       0       1  \n",
       "1FssTKSXO3JLiQc3udfK5I      0      0      0       0       0  \n",
       "0WWz2AaqxLoO0fa9ou6Fqc      0      1      0       0       0  \n",
       "5EeQ940KC0N4JCNLPIdRd1      0      0      0       0       0  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_X_train = all_X_train[audio_features]\n",
    "simple_X_test = all_X_test[audio_features]\n",
    "simple_X_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cbdf9da",
   "metadata": {},
   "source": [
    "* We will also export our basic datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "272bdfd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_X_train.to_csv(r'data/datasets/all/all_X_train.csv')\n",
    "all_X_test.to_csv(r'data/datasets/all/all_X_test.csv')\n",
    "imp_X_train.to_csv(r'data/datasets/important/imp_X_train.csv')\n",
    "imp_X_test.to_csv(r'data/datasets/important/imp_X_test.csv')\n",
    "y_train.to_csv(r'data/datasets/target/y_train.csv')\n",
    "y_test.to_csv(r'data/datasets/target/y_test.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f7ca1ac",
   "metadata": {
    "id": "rSjrqF8hqDFf"
   },
   "source": [
    "* Now we are ready to run our Machine Learning Algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42c76f50",
   "metadata": {
    "id": "tH5ItSy8HhG9"
   },
   "source": [
    "## K-Nearest Neighbors Regression\n",
    "* The first algorithm that we will use is a [KNN Regressor](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsRegressor.html).\n",
    "* Let's set up the scene.\n",
    "* We have to remember that our data must be scaled to use KNN algorithms.\n",
    "* So, we will create a <code>Pipeline</code>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "28356d80",
   "metadata": {
    "id": "QTrDLblIJFu1"
   },
   "outputs": [],
   "source": [
    "knn_regr = make_pipeline(StandardScaler(), KNeighborsRegressor(n_jobs=-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d76f4a07",
   "metadata": {
    "id": "EutKXL_6N9Os"
   },
   "source": [
    "* Let's create an initial solution with our simple dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f148058f",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "GiOjhF6EJX1O",
    "outputId": "9fb22f76-d9a9-440c-d821-e0a26c27e0d8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.15917493790344578"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_regr.fit(simple_X_train, y_train)\n",
    "simple_knn_pred = knn_regr.predict(simple_X_test)\n",
    "sklearn.metrics.mean_absolute_error(y_test, simple_knn_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "957d53a0",
   "metadata": {
    "id": "FR36WyogO1fQ"
   },
   "source": [
    "### All-Dataset\n",
    "* We will first use our dataset that contains all the variables.\n",
    "* To tune our algorithm we will use [Grid Search](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2b7e6429",
   "metadata": {
    "id": "TxlbnW9QRJwQ"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('standardscaler', StandardScaler()),\n",
       "                ('kneighborsregressor',\n",
       "                 KNeighborsRegressor(n_neighbors=10, p=1, weights='distance'))])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parameters = [\n",
    "  {\n",
    "   'kneighborsregressor__n_neighbors': [5, 10, 20, 50],\n",
    "   'kneighborsregressor__weights': ['uniform', 'distance'],\n",
    "   'kneighborsregressor__p': [1, 2]\n",
    "  }\n",
    "]\n",
    "\n",
    "knn_grid = GridSearchCV(knn_regr, parameters, cv=5, scoring='neg_mean_absolute_error', n_jobs=-1)\n",
    "knn_grid.fit(all_X_train, y_train)\n",
    "all_knn_r = knn_grid.best_estimator_\n",
    "all_knn_r"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04925747",
   "metadata": {
    "id": "a4TSirXJO5b5"
   },
   "source": [
    "* Let's see how well our Regressor operates on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "1a58921c",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JrAcFzcxOA2K",
    "outputId": "40c29c49-a4f5-439a-b4c8-dc6ab86b61c8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.12206381476315419"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_knn_pred = all_knn_r.predict(all_X_test)\n",
    "sklearn.metrics.mean_absolute_error(y_test, all_knn_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff0b1ee3",
   "metadata": {
    "id": "hdgmR_EwGi1I"
   },
   "source": [
    "* The result is much better than the simple model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd262715",
   "metadata": {
    "id": "YMrSG4GEe3wk"
   },
   "source": [
    "#### Weights\n",
    "* Here, an interesting idea would be to add **weights**, that are proportional to the impact that each variable has on valence.\n",
    "* The point here is that we want dimensions that have a significant impact on valence to affect the distance between the various points more.\n",
    "* A simple way to do that is to use the **standardized coefficients** of linear regression, as weights.\n",
    "*  But, we can't use the coefficients that we calculated before, because the dataset we used then, also contains our current test set.\n",
    "* So we have to calculate them again for our train dataset.\n",
    "* *Note: In order to simplify things, here we will include key_11 in the model, as we found that keys do not affect valence either way.*\n",
    "* To do all of this, we will create a custom scikit-learn **Transformer**.\n",
    "* It will weight each cell of every row, using the appropriate coefficient (simple mutiplication)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "dadebcab",
   "metadata": {
    "id": "xANkgvboimHH"
   },
   "outputs": [],
   "source": [
    "class WeightAdjustingTransformer(BaseEstimator, TransformerMixin):\n",
    "    \n",
    "    standarized_coeffs = []\n",
    "\n",
    "    def __init__(self):\n",
    "        pass\n",
    "    \n",
    "    def fit(self, X, y):\n",
    "        # turn X and y into appropriate dataframes\n",
    "        X_df = pd.DataFrame(X, columns=['x_' + str(i) for i in range(X.shape[1])])\n",
    "        y_df = pd.DataFrame(y)\n",
    "        y_df.index = X_df.index\n",
    "        y_df.columns = ['y']\n",
    "        # get the standarized coefficients\n",
    "        coeff_dict, pvalues_dict, sorted_metrics = self.get_standarized_coefficients(X_df.join(y_df), X_df.columns, 'y')\n",
    "        # put the correct values in the static variable <code>standarized_coeffs</code>.\n",
    "        WeightAdjustingTransformer.standarized_coeffs = np.array([coeff_dict[x] for x in X_df.columns])\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X, y=None):\n",
    "        return np.array([self.adjust_row(row) for row in X])\n",
    "    \n",
    "    @classmethod\n",
    "    def adjust_row(cls, row):\n",
    "        return np.array(row * cls.standarized_coeffs)\n",
    "    \n",
    "    @staticmethod\n",
    "    def get_standarized_coefficients(df, independent, target='valence'):\n",
    "      standarized_df = WeightAdjustingTransformer.get_standarized_dataframe(df, independent)\n",
    "      standarized_df = pd.concat((standarized_df, df[target]), axis=1) # add target column\n",
    "      standard_lr = WeightAdjustingTransformer.get_lr_res(standarized_df, independent, target)\n",
    "      coefs = standard_lr.params[1:] # don't include the Intercept\n",
    "      pvalues = standard_lr.pvalues[1:]\n",
    "      coef_dict = dict(zip(independent, coefs))\n",
    "      pvalues_dict = dict(zip(independent, pvalues))\n",
    "      return coef_dict, \\\n",
    "             pvalues_dict, \\\n",
    "             sorted(coef_dict, key=lambda x: abs(coef_dict[x]), reverse=True)\n",
    "\n",
    "    @staticmethod\n",
    "    def get_standarized_dataframe(df_source, columns):\n",
    "      scaler = StandardScaler()\n",
    "      scaler.fit(df_source[columns])\n",
    "      standarized_df = pd.DataFrame(scaler.transform(df_source[columns]), \n",
    "                                    index=df_source.index, \n",
    "                                    columns=columns)\n",
    "      return standarized_df\n",
    "    \n",
    "    @staticmethod\n",
    "    def get_lr_res(df, independent, target='valence'):\n",
    "      formula = f'{target} ~ ' + '+'.join(independent)\n",
    "      model = smf.ols(formula=formula, data=df)\n",
    "      return model.fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2abbbdd0",
   "metadata": {
    "id": "KmcAmEjijSFT"
   },
   "source": [
    "* We can now create our **Weighted KNN Regressor** pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "4984cc0d",
   "metadata": {
    "id": "jQXD7-OSjcO5"
   },
   "outputs": [],
   "source": [
    "weighted_knn_regr = make_pipeline(StandardScaler(), WeightAdjustingTransformer(), KNeighborsRegressor())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b974ae92",
   "metadata": {
    "id": "hG_cOHBpjmb-"
   },
   "source": [
    "* Let's see how well this new idea performs on the simple dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "id": "5ee77a90",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "VJmsdp6ejpu-",
    "outputId": "5d94ab57-fd9c-475d-9a1c-40c932f60281"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.14881601927447952"
      ]
     },
     "execution_count": 162,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weighted_knn_regr.fit(simple_X_train, y_train)\n",
    "simple_weighted_knn_pred = weighted_knn_regr.predict(simple_X_test)\n",
    "sklearn.metrics.mean_absolute_error(y_test, simple_weighted_knn_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "223a8b66",
   "metadata": {
    "id": "NVyh9LFgbOs2"
   },
   "source": [
    "* Seems to work better.\n",
    "* We will do Grid Search again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ec28a5be",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "i4KDVmvUbeBr",
    "outputId": "9b377c1b-682e-4681-8d7f-a438aed9194a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('standardscaler', StandardScaler()),\n",
       "                ('weightadjustingTransformer', WeightAdjustingTransformer()),\n",
       "                ('kneighborsregressor',\n",
       "                 KNeighborsRegressor(n_neighbors=10, p=1, weights='distance'))])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weighted_knn_grid = GridSearchCV(weighted_knn_regr, parameters, cv=5, scoring='neg_mean_absolute_error', n_jobs=-1)\n",
    "weighted_knn_grid.fit(all_X_train, y_train)\n",
    "all_weighted_knn_r = weighted_knn_grid.best_estimator_\n",
    "all_weighted_knn_r"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "063b7df7",
   "metadata": {
    "id": "htho6SYyAsIw"
   },
   "source": [
    "* We will evaluate on the test set again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "id": "5bb80886",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "nHw8voXSr8oj",
    "outputId": "5ff8ec94-b5ba-42f1-a24c-a19fd06a14c1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11269524398187614"
      ]
     },
     "execution_count": 165,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_weighted_knn_pred = all_weighted_knn_r.predict(all_X_test)\n",
    "sklearn.metrics.mean_absolute_error(y_test, all_weighted_knn_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d275ccaa",
   "metadata": {
    "id": "ZB-WCLpIAvMh"
   },
   "source": [
    "* The results are pretty good, realtively to the previous attempts."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9bf5dd1",
   "metadata": {
    "id": "v7mzHbSHA9YY"
   },
   "source": [
    "### Important-Dataset\n",
    "* We will use KNN for our Dataset with the \"important\" variables too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d6a85b55",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "OA9z39J_B_lk",
    "outputId": "3411be59-9677-498a-fc85-3cd1ffe0a32f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('standardscaler', StandardScaler()),\n",
       "                ('kneighborsregressor',\n",
       "                 KNeighborsRegressor(n_jobs=-1, n_neighbors=10, p=1,\n",
       "                                     weights='distance'))])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_grid.fit(imp_X_train, y_train)\n",
    "imp_knn_r = knn_grid.best_estimator_\n",
    "imp_knn_r"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9dcfd7f",
   "metadata": {
    "id": "w4jYFRxsCqgl"
   },
   "source": [
    "* Let's see how this one performs on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "id": "e7290eb3",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0neybO0zC0U6",
    "outputId": "9f911d1e-ca6d-477f-9803-30e3a298971b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11922329934561932"
      ]
     },
     "execution_count": 173,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imp_knn_pred = imp_knn_r.predict(imp_X_test)\n",
    "sklearn.metrics.mean_absolute_error(y_test, imp_knn_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86935379",
   "metadata": {
    "id": "naUWakjpNPZI"
   },
   "source": [
    "* The results here are better in comparison to the model with all the variables."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae34cb6e",
   "metadata": {
    "id": "tg8JgZLDC6xT"
   },
   "source": [
    "#### Weights\n",
    "* We will use weights for this dataset too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "5b93d195",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "A0MGrkrPDC5m",
    "outputId": "4d071c82-3de9-4c04-82cd-df189ea96d35"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('standardscaler', StandardScaler()),\n",
       "                ('weightadjustingtransformer', WeightAdjustingTransformer()),\n",
       "                ('kneighborsregressor',\n",
       "                 KNeighborsRegressor(n_neighbors=20, p=1, weights='distance'))])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weighted_knn_grid.fit(imp_X_train, y_train)\n",
    "imp_weighted_knn_r = weighted_knn_grid.best_estimator_\n",
    "imp_weighted_knn_r"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d19d08b2",
   "metadata": {
    "id": "BxexSRDaDdk4"
   },
   "source": [
    "* And we will test this regressor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "id": "a843e45a",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZBpxgGQsDi6b",
    "outputId": "831f89bb-3c1b-4373-b797-b91860863dd4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1067694786474835"
      ]
     },
     "execution_count": 179,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weighted_imp_knn_pred = imp_weighted_knn_r.predict(imp_X_test)\n",
    "sklearn.metrics.mean_absolute_error(y_test, weighted_imp_knn_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "010aea02",
   "metadata": {
    "id": "9dVlj3xHS2_A"
   },
   "source": [
    "* The results here are considerably better than any model that we tried before."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b371a65c",
   "metadata": {
    "id": "t_WhzFbWTDpN"
   },
   "source": [
    "## Linear Support Vector Regression.\n",
    "* The next Machine Lerning Algorithm that we will use is the [Linear Support Vector Regressor](https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVR.html#sklearn.svm.LinearSVR).\n",
    "* Let's create one.\n",
    "* We have to remember that SVMs are sensitive to scaling too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "2812e85c",
   "metadata": {
    "id": "GfhB5XRbTp_D"
   },
   "outputs": [],
   "source": [
    "lsvr = make_pipeline(StandardScaler(), LinearSVR(max_iter=10000))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1518ccf",
   "metadata": {
    "id": "UJMlieANUeZW"
   },
   "source": [
    "* We will create a base solution using our simple dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "806cd4b2",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "DONgeUjLUlE7",
    "outputId": "10a58aee-7da7-4eb1-a3e6-9bfca5e7aefd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.15661181039930386"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lsvr.fit(simple_X_train, y_train)\n",
    "simple_lsvr_pred = lsvr.predict(simple_X_test)\n",
    "sklearn.metrics.mean_absolute_error(y_test, simple_lsvr_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e85cebb",
   "metadata": {
    "id": "sqfnYz1JVmoR"
   },
   "source": [
    "* The first results of SVR are not particularly good.\n",
    "* We will see how it performs with our more detailed datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39e1f79a",
   "metadata": {
    "id": "stbchje8WEVT"
   },
   "source": [
    "### All-Dataset\n",
    "* Again we will use Grid Search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "6f90d94b",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UVEIFxLyWK21",
    "outputId": "ec29ced0-193f-4114-e591-b1a18144819a"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\georg\\appdata\\local\\programs\\python\\python39\\lib\\site-packages\\sklearn\\svm\\_base.py:1206: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('standardscaler', StandardScaler()),\n",
       "                ('linearsvr', LinearSVR(C=0.5, epsilon=0, max_iter=10000))])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parameters = [\n",
    "  {\n",
    "   'linearsvr__epsilon': [0, 0.1, 0.3],\n",
    "   'linearsvr__C': [0.5, 1, 2],\n",
    "  }\n",
    "]\n",
    "\n",
    "lsvr_grid = GridSearchCV(lsvr, parameters, cv=5, scoring='neg_mean_absolute_error', n_jobs=-1)\n",
    "lsvr_grid.fit(all_X_train, y_train)\n",
    "all_lsvr_r = lsvr_grid.best_estimator_\n",
    "all_lsvr_r"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77b4bc12",
   "metadata": {
    "id": "4mjeTRCBJ1aY"
   },
   "source": [
    "* Let's see the results on test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "78cd4441",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Ba_NQHiiYNFB",
    "outputId": "efbc12a7-23c8-4e0a-dc37-7d64b624f700"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.12218039898841296"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_lsvr_pred = all_lsvr_r.predict(all_X_test)\n",
    "sklearn.metrics.mean_absolute_error(y_test, all_lsvr_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ded81630",
   "metadata": {
    "id": "Y4Qc9gRWKFGf"
   },
   "source": [
    "#### Weights\n",
    "* As we have already created the <code>WeightAdjustingTransformer</code>. We can use it again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "db9c946b",
   "metadata": {
    "id": "UxM6QFw378dr"
   },
   "outputs": [],
   "source": [
    "weighted_lsvr = make_pipeline(StandardScaler(), WeightAdjustingTransformer(), LinearSVR())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4416ede",
   "metadata": {
    "id": "EqlXaZMKLYDW"
   },
   "source": [
    "* We will run Grid Search on this new LSVR and the dataset with all columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "ca152913",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "DnEJOKgE8IKH",
    "outputId": "c29f7a7c-020a-4a3f-ab14-f85b2a202900"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\georg\\appdata\\local\\programs\\python\\python39\\lib\\site-packages\\sklearn\\svm\\_base.py:1206: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('standardscaler', StandardScaler()),\n",
       "                ('weightadjustingtransformer', WeightAdjustingTransformer()),\n",
       "                ('linearsvr', LinearSVR(C=2, epsilon=0))])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weighted_lsvr_grid = GridSearchCV(weighted_lsvr, parameters, cv=5, scoring='neg_mean_absolute_error', n_jobs=-1)\n",
    "weighted_lsvr_grid.fit(all_X_train, y_train)\n",
    "all_weighted_lsvr_r = weighted_lsvr_grid.best_estimator_\n",
    "all_weighted_lsvr_r"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0382b11f",
   "metadata": {
    "id": "0BCh5OfkLrVQ"
   },
   "source": [
    "* Let's see the results on test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "66aed701",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0oaHjg-d8hXy",
    "outputId": "592cc500-5bf0-44c6-de44-10c698a082d4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.12194024193998934"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_weighted_lsvr_pred = all_weighted_lsvr_r.predict(all_X_test)\n",
    "sklearn.metrics.mean_absolute_error(y_test, all_weighted_lsvr_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efd41c78",
   "metadata": {
    "id": "nMfikudELwE9"
   },
   "source": [
    "* Here, the improvement is only very small."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83015999",
   "metadata": {
    "id": "A6cNeIaEL08Z"
   },
   "source": [
    "### Important-Dataset\n",
    "* We will apply the same procedure for the important dataset too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "c24ee05b",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "XECrM0eLZgfP",
    "outputId": "6a2d786e-6ac4-47f9-8b7a-99499a5eb29d"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\georg\\appdata\\local\\programs\\python\\python39\\lib\\site-packages\\sklearn\\svm\\_base.py:1206: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('standardscaler', StandardScaler()),\n",
       "                ('linearsvr', LinearSVR(C=1, epsilon=0, max_iter=10000))])"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lsvr_grid.fit(imp_X_train, y_train)\n",
    "imp_lsvr_r = lsvr_grid.best_estimator_\n",
    "imp_lsvr_r"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81033d06",
   "metadata": {
    "id": "B3NpDI3KMBqb"
   },
   "source": [
    "* And on test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "d79f75da",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gr9hCOmfZgfR",
    "outputId": "420bbbb3-0c5f-42c9-efe9-dc9d31b955a8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.12395255787752489"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imp_lsvr_pred = imp_lsvr_r.predict(imp_X_test)\n",
    "sklearn.metrics.mean_absolute_error(y_test, imp_lsvr_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d23caea9",
   "metadata": {
    "id": "nUN9CdTPMD32"
   },
   "source": [
    "* The results are worst in this case."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91df9e26",
   "metadata": {
    "id": "wlwWMfV1MGyQ"
   },
   "source": [
    "#### Weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "ab2ed6ca",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "onODFJg18qrD",
    "outputId": "e7ccc44d-9a87-485f-a51e-1feb39c6fb12"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('standardscaler', StandardScaler()),\n",
       "                ('weightadjustingtransformer', WeightAdjustingTransformer()),\n",
       "                ('linearsvr', LinearSVR(C=2, epsilon=0))])"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weighted_lsvr_grid.fit(imp_X_train, y_train)\n",
    "imp_weighted_lsvr_r = weighted_lsvr_grid.best_estimator_\n",
    "imp_weighted_lsvr_r"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce5d3bc7",
   "metadata": {
    "id": "flvIiRhkMJss"
   },
   "source": [
    "* Let's test again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "78f1b491",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "p56cCQPc8qrF",
    "outputId": "7c408c3d-8521-49e9-a15f-75246562b107"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.12392248046461102"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imp_weighted_lsvr_pred = imp_weighted_lsvr_r.predict(imp_X_test)\n",
    "sklearn.metrics.mean_absolute_error(y_test, imp_weighted_lsvr_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64683f87",
   "metadata": {
    "id": "OwSl_wi5MMYG"
   },
   "source": [
    "* The result here is identical, and worse than the model with all columns."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c16e92c",
   "metadata": {
    "id": "TGn9Budw9Ynm"
   },
   "source": [
    "## Tree-Based Algorithms\n",
    "* In this section, we will use tree-based algorithms, and specifically:\n",
    "  * [XGBoost](https://xgboost.readthedocs.io/en/stable/)\n",
    "  * [LightGBM](https://lightgbm.readthedocs.io/en/latest/)\n",
    "* The specific implementations are very strong and work well in practice."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bace139e",
   "metadata": {
    "id": "k9F-pMdJOMh1"
   },
   "source": [
    "## XGBoost\n",
    "* We will start by creating an XGBoost Regressor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "97a2cc8d",
   "metadata": {
    "id": "vdQo5HP_-wC1"
   },
   "outputs": [],
   "source": [
    "xgbr = xgb.XGBRegressor()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3d950cd",
   "metadata": {
    "id": "NPtopaEKQ0lw"
   },
   "source": [
    "* Let's get a base error estimation, using the simple dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "01cf5558",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "OG4r2uzvOoR5",
    "outputId": "0b487279-4834-4b29-f2f0-0741d3a9a8d6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[10:15:55] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.14064690888568981"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgbr.fit(simple_X_train, y_train)\n",
    "simple_xgb_pred = xgbr.predict(simple_X_test)\n",
    "sklearn.metrics.mean_absolute_error(y_test, simple_xgb_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4593d5d",
   "metadata": {
    "id": "_rBzwTMIRC5p"
   },
   "source": [
    "### All-Dataset\n",
    "* Here the runtime is very large, so instead of a simple Grid Search, we will use [Randomized Grid Search](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "209cc4f4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "D-YOAsdL9jgk",
    "outputId": "8abe06ba-d880-4758-a0ab-e296c02b3c2e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11:14:04] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "XGBRegressor(eta=0.3, gamma=0.5, max_depth=6)"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parameters = [\n",
    "  {\n",
    "   'booster': ['gbtree', 'dart'],\n",
    "   'eta': [0.1, 0.3, 0.6],\n",
    "   'gamma': [0.5, 3, 10, 20],\n",
    "   'max_depth': [3, 6, 10]\n",
    "  }\n",
    "]\n",
    "\n",
    "xgbr_random_grid = RandomizedSearchCV(xgbr, parameters, cv=5, scoring='neg_mean_absolute_error', n_jobs=-1)\n",
    "xgbr_random_grid.fit(all_X_train, y_train)\n",
    "all_xgbr_r = xgbr_random_grid.best_estimator_\n",
    "all_xgbr_r"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c6df113",
   "metadata": {
    "id": "LJdhWtKCLXcu"
   },
   "source": [
    "* Let's see how well this one performs at the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "cc674d6d",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ntbecKxe_D7n",
    "outputId": "0b595b9b-8cb9-48f6-c06e-3348e823886e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.09842982709213195"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_xgbr_pred = all_xgbr_r.predict(all_X_test)\n",
    "sklearn.metrics.mean_absolute_error(y_test, all_xgbr_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1f8d893",
   "metadata": {
    "id": "X3ISnzxPLdIh"
   },
   "source": [
    "* The results seem pretty good."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "975af9c3",
   "metadata": {
    "id": "5ihaD6auLjRi"
   },
   "source": [
    "### Important-Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "d19a61b8",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fXPSXPW3_SAO",
    "outputId": "545008d5-0748-434d-c2e3-e750b2d02214"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11:29:00] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "XGBRegressor(booster='dart', eta=0.1, gamma=0.5, max_depth=6)"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgbr_random_grid.fit(imp_X_train, y_train)\n",
    "imp_xgbr_r = xgbr_random_grid.best_estimator_\n",
    "imp_xgbr_r"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d49816a",
   "metadata": {
    "id": "GwPQSwrXLo1G"
   },
   "source": [
    "* Test again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "985cf0c7",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "iGSPbtvT_XXm",
    "outputId": "e5b1b793-c2a0-4476-95fa-3bc6386f7fac"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.10414892631559375"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imp_xgbr_pred = imp_xgbr_r.predict(imp_X_test)\n",
    "sklearn.metrics.mean_absolute_error(y_test, imp_xgbr_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfeb51fb",
   "metadata": {
    "id": "q64TJmb4LsKw"
   },
   "source": [
    "* Pretty good results again, but not as good as the dataset with all the variables."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4aefa03",
   "metadata": {
    "id": "Zoc9j3j4LzV8"
   },
   "source": [
    "## LightGBM\n",
    "* We will use a LightGBM Regressor next.\n",
    "* Let's create it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "91aa9e37",
   "metadata": {
    "id": "sk4Pcq_t_f7L"
   },
   "outputs": [],
   "source": [
    "lgbmr = lgb.LGBMRegressor()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91022ec0",
   "metadata": {
    "id": "85Sd2b_FMDbk"
   },
   "source": [
    "* Again we will get a base error for this estimator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "89124ea7",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "E31qltm0grHN",
    "outputId": "b7ae170b-edb5-4e62-ef6e-1160fbb1577c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1333431255291485"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lgbmr.fit(simple_X_train, y_train)\n",
    "simple_lgbm_pred = lgbmr.predict(simple_X_test)\n",
    "sklearn.metrics.mean_absolute_error(y_test, simple_lgbm_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ccbc7d3",
   "metadata": {
    "id": "ipT-uEOxMKKp"
   },
   "source": [
    "* This is the best we have seen yet, so the first indication is very promising.\n",
    "* Let's do our Grid Search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "id": "0a180e51",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9xBUOrBkwiqn",
    "outputId": "afb1d004-2abc-4df5-b0aa-c2ad73b2ca91"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LGBMRegressor(learning_rate=0.2, num_leaves=90)"
      ]
     },
     "execution_count": 163,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parameters = [\n",
    "  {\n",
    "   'boosting_type': ['gbdt', 'dart'],\n",
    "   'num_leaves': [10, 31, 60, 90],\n",
    "   'max_depth': [-1, 5 , 20],\n",
    "   'learning_rate': [0.05, 0.1, 0.2, 0.3]\n",
    "  }\n",
    "]\n",
    "\n",
    "lgbmr_grid = GridSearchCV(lgbmr, parameters, cv=5, scoring='neg_mean_absolute_error', n_jobs=-1)\n",
    "lgbmr_grid.fit(all_X_train, y_train)\n",
    "all_lgbmr_r = lgbmr_grid.best_estimator_\n",
    "all_lgbmr_r"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d81c9eef",
   "metadata": {
    "id": "XzdyCx-vMsR2"
   },
   "source": [
    "* And have a look at how well it performs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "id": "80722ee4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Rj5Ox6rrNLYj",
    "outputId": "3b75f1c8-2ef5-4d25-9b2d-571f09f7b56f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.09164649690604323"
      ]
     },
     "execution_count": 255,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_lgbmr_pred = all_lgbmr_r.predict(all_X_test)\n",
    "sklearn.metrics.mean_absolute_error(y_test, all_lgbmr_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "216943bb",
   "metadata": {
    "id": "ppvPbXMcNnxP"
   },
   "source": [
    "* Wow! this is much lower than any value found previously.\n",
    "* Is there any chance that we can **tune it a bit more**?\n",
    "* Here the idea is to give our Regressor a bit **more estimators** to fit. But in order to get advantage of it, it is necessary to **increase the number of leaves** and **reduce the learning rate**.\n",
    "* After experimenting a bit we concluded that some parameters that work relatively well are:\n",
    "  * <code>learning_rate</code>: 0.05\n",
    "  * <code>num_leaves</code>: 200\n",
    "  * <code>n_estimators</code>: 1500\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "43a6eaa1",
   "metadata": {
    "id": "i390Hyj5PRuQ"
   },
   "outputs": [],
   "source": [
    "fine_all_lgbmr_r = lgb.LGBMRegressor(learning_rate=0.05, num_leaves=200, n_estimators=1500)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a9bf8b4",
   "metadata": {
    "id": "W5lr0e83PSi3"
   },
   "source": [
    "* So, let's check its performance. \n",
    "* First, we will do a cross validation on the train test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "d252b2c0",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "uIGIlhTEPFCf",
    "outputId": "b6f1a2e1-75ab-447f-e12f-c6f0e9854793"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results: [0.08575149 0.08835177 0.08765204 0.08921213 0.08784873], Average: 0.08776323228900765\n"
     ]
    }
   ],
   "source": [
    "cv_results = cross_validate(fine_all_lgbmr_r, all_X_train, y_train, cv=5, scoring='neg_mean_absolute_error')\n",
    "print('Results: {0}, Average: {1}'.format(-cv_results['test_score'], -cv_results['test_score'].mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adf3f491",
   "metadata": {
    "id": "b6nDmmQ7RXfz"
   },
   "source": [
    "* In comparison let's see how well our initial LBGM Regressor performed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "3bc18553",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "teeW2bhCRiFy",
    "outputId": "59836133-a29a-4aa9-8377-55d47bc12ab3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results: [0.09085612 0.09369718 0.0933885  0.09414988 0.09293705], Average: 0.09300574470905838\n"
     ]
    }
   ],
   "source": [
    "cv_results = cross_validate(all_lgbmr_r, all_X_train, y_train, cv=5, scoring='neg_mean_absolute_error')\n",
    "print('Results: {0}, Average: {1}'.format(-cv_results['test_score'], -cv_results['test_score'].mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d403bb7",
   "metadata": {
    "id": "nyZeX3qPRvrZ"
   },
   "source": [
    "* The results of the fine-tuned are considerably better.\n",
    "* And finally, let's see the perfomance on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "a756ddbb",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FGmtCuKXR4_7",
    "outputId": "1772dfe9-53bb-4dc9-e1d6-a8d046904fb6"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.08537355791917575"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fine_all_lgbmr_r.fit(all_X_train, y_train)\n",
    "fine_all_lgbmr_pred = fine_all_lgbmr_r.predict(all_X_test)\n",
    "sklearn.metrics.mean_absolute_error(y_test, fine_all_lgbmr_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bf74ba5",
   "metadata": {
    "id": "dE5anJNxSEZ6"
   },
   "source": [
    "* The results are the best we have seen."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f9f8971",
   "metadata": {
    "id": "vP-isj3KSILx"
   },
   "source": [
    "### Important-Dataset\n",
    "* Is there any chance we can get better results with the important dataset?\n",
    "* Let's perform Grid Search again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 278,
   "id": "f5c1f593",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YcosVwVD_m_4",
    "outputId": "1c82c6a8-3dcb-499f-9d9a-e93947106584"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LGBMRegressor(max_depth=20, num_leaves=90)"
      ]
     },
     "execution_count": 278,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lgbmr_grid.fit(imp_X_train, y_train)\n",
    "imp_lgbmr_r = lgbmr_grid.best_estimator_\n",
    "imp_lgbmr_r"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5d336e3",
   "metadata": {
    "id": "MAfJhsVVgQ8C"
   },
   "source": [
    "* Let's test the performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 280,
   "id": "463154bc",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1XCGRVtMgOvX",
    "outputId": "13f2fbb4-668a-4647-8898-c070820cd7cd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.09876587876028549"
      ]
     },
     "execution_count": 280,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imp_lgbmr_pred = imp_lgbmr_r.predict(imp_X_test)\n",
    "sklearn.metrics.mean_absolute_error(y_test, imp_lgbmr_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c51736d",
   "metadata": {
    "id": "2nf9YxMDgfk0"
   },
   "source": [
    "* The error we get here is singificantly higher than the previous model, so we won't try to tune it even more."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee1735fd",
   "metadata": {
    "id": "I9So4cTzgrsa"
   },
   "source": [
    "### Conlusions - Non Neural Network Machine Learning\n",
    "* The best results were achieved with the **LGBM Regressor**.\n",
    "* The Minimum Mean Absolute Error for the test set was: **0.0853**.\n",
    "* But let's dive a little deeper and try to interpret how this results were calculated.\n",
    "* To do that, we will use the **feature importances** of our best regressor. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "14c229a1",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "N6vLQ95GJVGi",
    "outputId": "59cefc25-a1e0-41c7-83ff-ce087195c5c2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. feature 9 duration_ms (4058.000000)\n",
      "2. feature 8 tempo (3534.000000)\n",
      "3. feature 7 liveness (3388.000000)\n",
      "4. feature 4 speechiness (3191.000000)\n",
      "5. feature 134 duration_std_mid (3166.000000)\n",
      "6. feature 78 duration_std_start (2993.000000)\n",
      "7. feature 0 danceability (2938.000000)\n",
      "8. feature 108 pitches_2_std_mid (2917.000000)\n",
      "9. feature 79 pitches_1_means_mid (2853.000000)\n",
      "10. feature 97 timbre_7_means_mid (2849.000000)\n"
     ]
    }
   ],
   "source": [
    "importances = fine_all_lgbmr_r.feature_importances_\n",
    "indices = np.argsort(importances)[::-1]\n",
    "feature_cols = all_X_test.columns\n",
    "    \n",
    "for f in range(10):\n",
    "    print(\"%d. feature %d %s (%f)\" % (f + 1, indices[f], feature_cols[indices[f]], importances[indices[f]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8044fe6",
   "metadata": {
    "id": "xfFPatyIk8CP"
   },
   "source": [
    "* The three most important predictors according to the feature importances are:\n",
    "  * <code>duration_ms</code>\n",
    "  * <code>tempo</code>\n",
    "  * <code>liveness</code>\n",
    "* This may be a little surprising, but we have to remember that each of these is combined with all the others to get good results.\n",
    "* We can't be sure, but for example we can see that <code>duration_std_mid</code> and <code>duration_std_start</code> are also very high. So the combination of the duration of the  song with the standard deviation of the duration of each segments, may be important for the calculation of valence.\n",
    "* Also, <code>liveness</code> and <code>speechiness</code> are columns that we didn't include in our important dataset, which is something that may explain the better results we get with this one.\n",
    "* Lastly, *pitch 1* appears twice in the top feature importances, which shows us that it must be one of the most important features that we get from the deep analysis.\n",
    "* Additionally, <code>loudness</code> does not appear a single time here, which may show that does not affect valence after all.\n",
    "* In general a more high level interpretation of the feature importances gives an indication that *valence* has a lot to do with the rhythm of the song (variance of the duration of the segments, tempo, danceability, all relate to the rhythm)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}