source-code/pandas/copy_on_write.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e89344bf-36d0-4af0-bc1b-85a461d348cb",
   "metadata": {},
   "source": [
    "## Requirments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9550bae2-79e5-4db6-b8eb-4b64dbc1b4e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "712b5866-99b7-43ac-8c80-ec209938796b",
   "metadata": {},
   "source": [
    "## Shared or not shared?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d524ce09-82c7-4f40-8fea-9ec3439345e3",
   "metadata": {},
   "source": [
    "When creating a new dataframe out of an existing dataframe, is column data shared or not?\n",
    "\n",
    "The answer to that question seems to depend on the version of `pandas`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea08f1e0-78c1-463c-994c-92066363b006",
   "metadata": {},
   "source": [
    "## Pandas 2.x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df04d393-9f34-45c0-99e6-fba4d8bbf3d5",
   "metadata": {},
   "source": [
    "To see what happens when you modify a dataframe from which another is derived, you can start by creating a large dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "520bc27d-5d55-46dc-a7bf-5b9b12162b9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.DataFrame({\n",
    "    'column1': np.random.uniform(-1_000.0, 1_000.0, size=100_000),\n",
    "    'column2': np.random.uniform(-1_000.0, 1_000.0, size=100_000),\n",
    "    'column3': np.random.uniform(-1_000, 1_000, size=100_000).astype(np.int64),\n",
    "    'column4': np.random.uniform(-1_000, 1_000, size=100_000).astype(np.int64),\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "432772e9-3031-4001-81e3-c346cb7f9c76",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 100000 entries, 0 to 99999\n",
      "Data columns (total 4 columns):\n",
      " #   Column   Non-Null Count   Dtype  \n",
      "---  ------   --------------   -----  \n",
      " 0   column1  100000 non-null  float64\n",
      " 1   column2  100000 non-null  float64\n",
      " 2   column3  100000 non-null  int64  \n",
      " 3   column4  100000 non-null  int64  \n",
      "dtypes: float64(2), int64(2)\n",
      "memory usage: 3.1 MB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf2c9b41-07b0-41d5-b5ae-22a43828eaac",
   "metadata": {},
   "source": [
    "The data has the statistical properties you would expect from a uniform distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "eb6f4dad-6df3-493e-869f-204d06b0cc7d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>column1</th>\n",
       "      <th>column2</th>\n",
       "      <th>column3</th>\n",
       "      <th>column4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>100000.000000</td>\n",
       "      <td>100000.000000</td>\n",
       "      <td>100000.000000</td>\n",
       "      <td>100000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>4.225888</td>\n",
       "      <td>1.284407</td>\n",
       "      <td>-2.830820</td>\n",
       "      <td>0.143920</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>577.759840</td>\n",
       "      <td>577.063194</td>\n",
       "      <td>577.528168</td>\n",
       "      <td>576.729627</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-999.977694</td>\n",
       "      <td>-999.908941</td>\n",
       "      <td>-999.000000</td>\n",
       "      <td>-999.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-495.491551</td>\n",
       "      <td>-497.783061</td>\n",
       "      <td>-506.000000</td>\n",
       "      <td>-499.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>4.748510</td>\n",
       "      <td>2.103870</td>\n",
       "      <td>-4.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>505.160586</td>\n",
       "      <td>500.502871</td>\n",
       "      <td>497.000000</td>\n",
       "      <td>498.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>999.992940</td>\n",
       "      <td>999.947904</td>\n",
       "      <td>999.000000</td>\n",
       "      <td>999.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             column1        column2        column3        column4\n",
       "count  100000.000000  100000.000000  100000.000000  100000.000000\n",
       "mean        4.225888       1.284407      -2.830820       0.143920\n",
       "std       577.759840     577.063194     577.528168     576.729627\n",
       "min      -999.977694    -999.908941    -999.000000    -999.000000\n",
       "25%      -495.491551    -497.783061    -506.000000    -499.000000\n",
       "50%         4.748510       2.103870      -4.000000       0.000000\n",
       "75%       505.160586     500.502871     497.000000     498.000000\n",
       "max       999.992940     999.947904     999.000000     999.000000"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39f69e87-dd89-4451-9d62-1172dcefac6f",
   "metadata": {},
   "source": [
    "When you create a new dataframe with half the number of rows as the original dataset, the question is whether or not the two dataframes share data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "1a43e7eb-a7ad-4089-95ba-15879f5920ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "data2 = data.iloc[:50_000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "1af4b0bc-81eb-4dab-9a65-fb5f71d004c2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>column1</th>\n",
       "      <th>column2</th>\n",
       "      <th>column3</th>\n",
       "      <th>column4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.00000</td>\n",
       "      <td>50000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>4.828118</td>\n",
       "      <td>0.964712</td>\n",
       "      <td>-2.11148</td>\n",
       "      <td>-3.656300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>577.936131</td>\n",
       "      <td>577.971021</td>\n",
       "      <td>577.96331</td>\n",
       "      <td>576.181292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-999.944283</td>\n",
       "      <td>-999.897600</td>\n",
       "      <td>-999.00000</td>\n",
       "      <td>-999.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-493.817751</td>\n",
       "      <td>-500.003775</td>\n",
       "      <td>-504.00000</td>\n",
       "      <td>-503.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>4.462345</td>\n",
       "      <td>3.577384</td>\n",
       "      <td>-6.00000</td>\n",
       "      <td>-2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>503.114598</td>\n",
       "      <td>500.886860</td>\n",
       "      <td>501.00000</td>\n",
       "      <td>494.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>999.964569</td>\n",
       "      <td>999.864196</td>\n",
       "      <td>999.00000</td>\n",
       "      <td>999.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            column1       column2      column3       column4\n",
       "count  50000.000000  50000.000000  50000.00000  50000.000000\n",
       "mean       4.828118      0.964712     -2.11148     -3.656300\n",
       "std      577.936131    577.971021    577.96331    576.181292\n",
       "min     -999.944283   -999.897600   -999.00000   -999.000000\n",
       "25%     -493.817751   -500.003775   -504.00000   -503.000000\n",
       "50%        4.462345      3.577384     -6.00000     -2.000000\n",
       "75%      503.114598    500.886860    501.00000    494.000000\n",
       "max      999.964569    999.864196    999.00000    999.000000"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data2.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9325195d-d768-45a9-91e6-c777ae03c30b",
   "metadata": {},
   "source": [
    "To ascertain this, you can modify one of the dataframes, say `data` in place, and check whether the other dataframe is affected.  For insstance, yu can clip the values of the first column to the interval $[-500, 500]$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "65dfb904-0b0a-4b4a-baab-77011a840910",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_565/3787905307.py:1: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.\n",
      "The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.\n",
      "\n",
      "For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.\n",
      "\n",
      "\n",
      "  data['column1'].clip(-500.0, 500.0, inplace=True)\n"
     ]
    }
   ],
   "source": [
    "data['column1'].clip(-500.0, 500.0, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "cbeb2db9-7018-4cac-9dd4-2880d2f7a214",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>column1</th>\n",
       "      <th>column2</th>\n",
       "      <th>column3</th>\n",
       "      <th>column4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.00000</td>\n",
       "      <td>50000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>2.605923</td>\n",
       "      <td>0.964712</td>\n",
       "      <td>-2.11148</td>\n",
       "      <td>-3.656300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>408.136559</td>\n",
       "      <td>577.971021</td>\n",
       "      <td>577.96331</td>\n",
       "      <td>576.181292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-500.000000</td>\n",
       "      <td>-999.897600</td>\n",
       "      <td>-999.00000</td>\n",
       "      <td>-999.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-493.817751</td>\n",
       "      <td>-500.003775</td>\n",
       "      <td>-504.00000</td>\n",
       "      <td>-503.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>4.462345</td>\n",
       "      <td>3.577384</td>\n",
       "      <td>-6.00000</td>\n",
       "      <td>-2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>500.000000</td>\n",
       "      <td>500.886860</td>\n",
       "      <td>501.00000</td>\n",
       "      <td>494.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>500.000000</td>\n",
       "      <td>999.864196</td>\n",
       "      <td>999.00000</td>\n",
       "      <td>999.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            column1       column2      column3       column4\n",
       "count  50000.000000  50000.000000  50000.00000  50000.000000\n",
       "mean       2.605923      0.964712     -2.11148     -3.656300\n",
       "std      408.136559    577.971021    577.96331    576.181292\n",
       "min     -500.000000   -999.897600   -999.00000   -999.000000\n",
       "25%     -493.817751   -500.003775   -504.00000   -503.000000\n",
       "50%        4.462345      3.577384     -6.00000     -2.000000\n",
       "75%      500.000000    500.886860    501.00000    494.000000\n",
       "max      500.000000    999.864196    999.00000    999.000000"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data2.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e50a51a8-3145-4cbf-9103-665ac5efe9f2",
   "metadata": {},
   "source": [
    "Indeed, the first column of `data2` was affected, its minimum and maximem values aore no longer approximately -1000 and 1000 as before, but rather -500 and 500 respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82d41844-34c4-497e-b9d0-1406d2bc7aa4",
   "metadata": {},
   "source": [
    "This may surprise you, and is of course a potential source of subtle bugs.  It is a trade-off, on the one hand sharing data between related dataframes saves memory, but also increases performance, since copying data is expensive in terms of execution time.  On the other hand, it means you perceive \"spooky action at a distance\" since operations on one dataframe may affect a related dataframe."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb1848d4-6211-447b-95a8-3dd0732d8ebe",
   "metadata": {},
   "source": [
    "The developers of pandas want to ensure that an operation on a dataframe never affects another dataframe and have introduced Copy-on-Write in pandas 1.5.x.  It is a \"mode\" that can be set via pandas `options` module.  Pandas 2.x will generate some warnings for obvious cases when Copy-on-Write is violated, so that you can adapt your code accordingly.\n",
    "\n",
    "The warning displayed when you executed the `clip` operation in-place is an illustration of that."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "400a9a99-fc50-4290-b411-70fa73af72e4",
   "metadata": {},
   "source": [
    "## Pandas 3.x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df6e8f82-aa76-40cf-8d88-5d82a3f5cf16",
   "metadata": {},
   "source": [
    "In pandas 3.x, Copy-on-Write will be the only mode that is available, it will ensure that only a single dataframe or series is affected by an operation.  When using pandas 2.x, you can optionally switch on this semantics already."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "b4cff949-c4aa-4f7d-b1e4-b4b78bf0284b",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.options.mode.copy_on_write = True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b9d5071-862d-46e4-9dbe-912b392345b7",
   "metadata": {},
   "source": [
    "You can replay the same scenario as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "6bb4fec9-0623-4482-8ebd-9077723956e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.DataFrame({\n",
    "    'column1': np.random.uniform(-1_000.0, 1_000.0, size=100_000),\n",
    "    'column2': np.random.uniform(-1_000.0, 1_000.0, size=100_000),\n",
    "    'column3': np.random.uniform(-1_000, 1_000, size=100_000).astype(np.int64),\n",
    "    'column4': np.random.uniform(-1_000, 1_000, size=100_000).astype(np.int64),\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "f005ad9f-3b19-40bf-b01a-4f7fd8f4d024",
   "metadata": {},
   "outputs": [],
   "source": [
    "data2 = data.iloc[:50_000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "91499993-8461-437d-b7b1-7f0caf20d7d6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>column1</th>\n",
       "      <th>column2</th>\n",
       "      <th>column3</th>\n",
       "      <th>column4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.695724</td>\n",
       "      <td>-3.040382</td>\n",
       "      <td>4.001460</td>\n",
       "      <td>0.511100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>578.627560</td>\n",
       "      <td>576.453798</td>\n",
       "      <td>576.244217</td>\n",
       "      <td>578.301376</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-999.983952</td>\n",
       "      <td>-999.968792</td>\n",
       "      <td>-999.000000</td>\n",
       "      <td>-999.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-502.160012</td>\n",
       "      <td>-500.333306</td>\n",
       "      <td>-492.250000</td>\n",
       "      <td>-498.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>4.057258</td>\n",
       "      <td>-4.377464</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>502.159214</td>\n",
       "      <td>494.614704</td>\n",
       "      <td>499.000000</td>\n",
       "      <td>500.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>999.998666</td>\n",
       "      <td>999.913716</td>\n",
       "      <td>999.000000</td>\n",
       "      <td>999.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            column1       column2       column3       column4\n",
       "count  50000.000000  50000.000000  50000.000000  50000.000000\n",
       "mean       1.695724     -3.040382      4.001460      0.511100\n",
       "std      578.627560    576.453798    576.244217    578.301376\n",
       "min     -999.983952   -999.968792   -999.000000   -999.000000\n",
       "25%     -502.160012   -500.333306   -492.250000   -498.000000\n",
       "50%        4.057258     -4.377464      3.000000      0.000000\n",
       "75%      502.159214    494.614704    499.000000    500.250000\n",
       "max      999.998666    999.913716    999.000000    999.000000"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data2.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "abf7f30b-2bd0-4b3b-94fd-f9b6f183b26b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_565/3016868282.py:1: ChainedAssignmentError: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.\n",
      "When using the Copy-on-Write mode, such inplace method never works to update the original DataFrame or Series, because the intermediate object on which we are setting values always behaves as a copy.\n",
      "\n",
      "For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' instead, to perform the operation inplace on the original object.\n",
      "\n",
      "\n",
      "  data['column2'].clip(-500.0, 500.0, inplace=True)\n"
     ]
    }
   ],
   "source": [
    "data['column2'].clip(-500.0, 500.0, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13934be0-c460-4a3d-ae3b-ceb00108f868",
   "metadata": {},
   "source": [
    "This time around, thanks to setting the `copy_on_write` mode to `True`, an error is raised, and you can verify that `data2` is unaffected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "34e80602-a9fa-44d2-9fbb-3b6d4bd6a3d3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>column1</th>\n",
       "      <th>column2</th>\n",
       "      <th>column3</th>\n",
       "      <th>column4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.695724</td>\n",
       "      <td>-3.040382</td>\n",
       "      <td>4.001460</td>\n",
       "      <td>0.511100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>578.627560</td>\n",
       "      <td>576.453798</td>\n",
       "      <td>576.244217</td>\n",
       "      <td>578.301376</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-999.983952</td>\n",
       "      <td>-999.968792</td>\n",
       "      <td>-999.000000</td>\n",
       "      <td>-999.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-502.160012</td>\n",
       "      <td>-500.333306</td>\n",
       "      <td>-492.250000</td>\n",
       "      <td>-498.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>4.057258</td>\n",
       "      <td>-4.377464</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>502.159214</td>\n",
       "      <td>494.614704</td>\n",
       "      <td>499.000000</td>\n",
       "      <td>500.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>999.998666</td>\n",
       "      <td>999.913716</td>\n",
       "      <td>999.000000</td>\n",
       "      <td>999.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            column1       column2       column3       column4\n",
       "count  50000.000000  50000.000000  50000.000000  50000.000000\n",
       "mean       1.695724     -3.040382      4.001460      0.511100\n",
       "std      578.627560    576.453798    576.244217    578.301376\n",
       "min     -999.983952   -999.968792   -999.000000   -999.000000\n",
       "25%     -502.160012   -500.333306   -492.250000   -498.000000\n",
       "50%        4.057258     -4.377464      3.000000      0.000000\n",
       "75%      502.159214    494.614704    499.000000    500.250000\n",
       "max      999.998666    999.913716    999.000000    999.000000"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data2.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20b2b5b9-423a-4e71-88f0-3175c445606f",
   "metadata": {},
   "source": [
    "Of course, the orginal dataframe `data` is also unmodified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "acc94632-2ed2-4dc1-a067-6d9518044c7b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>column1</th>\n",
       "      <th>column2</th>\n",
       "      <th>column3</th>\n",
       "      <th>column4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>100000.000000</td>\n",
       "      <td>100000.000000</td>\n",
       "      <td>100000.00000</td>\n",
       "      <td>100000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.623964</td>\n",
       "      <td>-3.764614</td>\n",
       "      <td>1.34378</td>\n",
       "      <td>1.478400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>577.971329</td>\n",
       "      <td>577.167935</td>\n",
       "      <td>575.56337</td>\n",
       "      <td>577.826276</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-999.983952</td>\n",
       "      <td>-999.980827</td>\n",
       "      <td>-999.00000</td>\n",
       "      <td>-999.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-499.179172</td>\n",
       "      <td>-503.072353</td>\n",
       "      <td>-497.00000</td>\n",
       "      <td>-497.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.133260</td>\n",
       "      <td>-5.691740</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>502.454035</td>\n",
       "      <td>495.773629</td>\n",
       "      <td>497.00000</td>\n",
       "      <td>501.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>999.998666</td>\n",
       "      <td>999.985790</td>\n",
       "      <td>999.00000</td>\n",
       "      <td>999.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             column1        column2       column3        column4\n",
       "count  100000.000000  100000.000000  100000.00000  100000.000000\n",
       "mean        1.623964      -3.764614       1.34378       1.478400\n",
       "std       577.971329     577.167935     575.56337     577.826276\n",
       "min      -999.983952    -999.980827    -999.00000    -999.000000\n",
       "25%      -499.179172    -503.072353    -497.00000    -497.000000\n",
       "50%         3.133260      -5.691740       1.00000       2.000000\n",
       "75%       502.454035     495.773629     497.00000     501.000000\n",
       "max       999.998666     999.985790     999.00000     999.000000"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "890d7709-3e75-4ebb-a13e-f61f0f2dd347",
   "metadata": {},
   "source": [
    "Following the advice of the warning/error message, you can execute the `clip` operation on the dataframe, specifying the clip values for each column, where `None` will not clip any values for that column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "1b86dd08-d8de-4b03-a363-80d18201b4da",
   "metadata": {},
   "outputs": [],
   "source": [
    "data.clip([-500.0, *[None]*3], [500.0, *[None]*3], inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3ff18cd-b795-49a2-a63c-d4f50b52fb3b",
   "metadata": {},
   "source": [
    "You can verify that `data2` is unaffected, as intended."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "dea886ac-e1a2-4903-b550-e73a2be70507",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>column1</th>\n",
       "      <th>column2</th>\n",
       "      <th>column3</th>\n",
       "      <th>column4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.695724</td>\n",
       "      <td>-3.040382</td>\n",
       "      <td>4.001460</td>\n",
       "      <td>0.511100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>578.627560</td>\n",
       "      <td>576.453798</td>\n",
       "      <td>576.244217</td>\n",
       "      <td>578.301376</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-999.983952</td>\n",
       "      <td>-999.968792</td>\n",
       "      <td>-999.000000</td>\n",
       "      <td>-999.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-502.160012</td>\n",
       "      <td>-500.333306</td>\n",
       "      <td>-492.250000</td>\n",
       "      <td>-498.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>4.057258</td>\n",
       "      <td>-4.377464</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>502.159214</td>\n",
       "      <td>494.614704</td>\n",
       "      <td>499.000000</td>\n",
       "      <td>500.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>999.998666</td>\n",
       "      <td>999.913716</td>\n",
       "      <td>999.000000</td>\n",
       "      <td>999.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            column1       column2       column3       column4\n",
       "count  50000.000000  50000.000000  50000.000000  50000.000000\n",
       "mean       1.695724     -3.040382      4.001460      0.511100\n",
       "std      578.627560    576.453798    576.244217    578.301376\n",
       "min     -999.983952   -999.968792   -999.000000   -999.000000\n",
       "25%     -502.160012   -500.333306   -492.250000   -498.000000\n",
       "50%        4.057258     -4.377464      3.000000      0.000000\n",
       "75%      502.159214    494.614704    499.000000    500.250000\n",
       "max      999.998666    999.913716    999.000000    999.000000"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data2.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d7b4146-61e3-4edd-9df5-4c7cc395cc86",
   "metadata": {},
   "source": [
    "Of course, the values in the first column of `data` are clipped as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "c73063c8-4e9d-42ba-9804-1286b769ad54",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>column1</th>\n",
       "      <th>column2</th>\n",
       "      <th>column3</th>\n",
       "      <th>column4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>100000.000000</td>\n",
       "      <td>100000.000000</td>\n",
       "      <td>100000.00000</td>\n",
       "      <td>100000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.174175</td>\n",
       "      <td>-3.764614</td>\n",
       "      <td>1.34378</td>\n",
       "      <td>1.478400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>408.414807</td>\n",
       "      <td>577.167935</td>\n",
       "      <td>575.56337</td>\n",
       "      <td>577.826276</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-500.000000</td>\n",
       "      <td>-999.980827</td>\n",
       "      <td>-999.00000</td>\n",
       "      <td>-999.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-499.179172</td>\n",
       "      <td>-503.072353</td>\n",
       "      <td>-497.00000</td>\n",
       "      <td>-497.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.133260</td>\n",
       "      <td>-5.691740</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>500.000000</td>\n",
       "      <td>495.773629</td>\n",
       "      <td>497.00000</td>\n",
       "      <td>501.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>500.000000</td>\n",
       "      <td>999.985790</td>\n",
       "      <td>999.00000</td>\n",
       "      <td>999.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             column1        column2       column3        column4\n",
       "count  100000.000000  100000.000000  100000.00000  100000.000000\n",
       "mean        1.174175      -3.764614       1.34378       1.478400\n",
       "std       408.414807     577.167935     575.56337     577.826276\n",
       "min      -500.000000    -999.980827    -999.00000    -999.000000\n",
       "25%      -499.179172    -503.072353    -497.00000    -497.000000\n",
       "50%         3.133260      -5.691740       1.00000       2.000000\n",
       "75%       500.000000     495.773629     497.00000     501.000000\n",
       "max       500.000000     999.985790     999.00000     999.000000"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f2be0a1-c90f-4d62-a19d-cb5307c3c2ff",
   "metadata": {},
   "source": [
    "Alternatively, the values can also be clipped as follows for the second column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "1151d8a4-db49-4524-b7f3-bd38866e3d5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "data['column2'] = data['column2'].clip(-500.0, 500.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "bd8ce0d1-73cb-4982-b4fe-65e7d2144943",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>column1</th>\n",
       "      <th>column2</th>\n",
       "      <th>column3</th>\n",
       "      <th>column4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>100000.000000</td>\n",
       "      <td>100000.000000</td>\n",
       "      <td>100000.00000</td>\n",
       "      <td>100000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.174175</td>\n",
       "      <td>-2.876739</td>\n",
       "      <td>1.34378</td>\n",
       "      <td>1.478400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>408.414807</td>\n",
       "      <td>408.107880</td>\n",
       "      <td>575.56337</td>\n",
       "      <td>577.826276</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-500.000000</td>\n",
       "      <td>-500.000000</td>\n",
       "      <td>-999.00000</td>\n",
       "      <td>-999.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-499.179172</td>\n",
       "      <td>-500.000000</td>\n",
       "      <td>-497.00000</td>\n",
       "      <td>-497.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.133260</td>\n",
       "      <td>-5.691740</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>500.000000</td>\n",
       "      <td>495.773629</td>\n",
       "      <td>497.00000</td>\n",
       "      <td>501.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>500.000000</td>\n",
       "      <td>500.000000</td>\n",
       "      <td>999.00000</td>\n",
       "      <td>999.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             column1        column2       column3        column4\n",
       "count  100000.000000  100000.000000  100000.00000  100000.000000\n",
       "mean        1.174175      -2.876739       1.34378       1.478400\n",
       "std       408.414807     408.107880     575.56337     577.826276\n",
       "min      -500.000000    -500.000000    -999.00000    -999.000000\n",
       "25%      -499.179172    -500.000000    -497.00000    -497.000000\n",
       "50%         3.133260      -5.691740       1.00000       2.000000\n",
       "75%       500.000000     495.773629     497.00000     501.000000\n",
       "max       500.000000     500.000000     999.00000     999.000000"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38e046dc-9f49-4a12-9304-72fb948a550f",
   "metadata": {},
   "source": [
    "Dataframe `data2` is unaffected as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "bea974f5-367b-4432-893f-a33a449dda6f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>column1</th>\n",
       "      <th>column2</th>\n",
       "      <th>column3</th>\n",
       "      <th>column4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.695724</td>\n",
       "      <td>-3.040382</td>\n",
       "      <td>4.001460</td>\n",
       "      <td>0.511100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>578.627560</td>\n",
       "      <td>576.453798</td>\n",
       "      <td>576.244217</td>\n",
       "      <td>578.301376</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-999.983952</td>\n",
       "      <td>-999.968792</td>\n",
       "      <td>-999.000000</td>\n",
       "      <td>-999.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-502.160012</td>\n",
       "      <td>-500.333306</td>\n",
       "      <td>-492.250000</td>\n",
       "      <td>-498.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>4.057258</td>\n",
       "      <td>-4.377464</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>502.159214</td>\n",
       "      <td>494.614704</td>\n",
       "      <td>499.000000</td>\n",
       "      <td>500.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>999.998666</td>\n",
       "      <td>999.913716</td>\n",
       "      <td>999.000000</td>\n",
       "      <td>999.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            column1       column2       column3       column4\n",
       "count  50000.000000  50000.000000  50000.000000  50000.000000\n",
       "mean       1.695724     -3.040382      4.001460      0.511100\n",
       "std      578.627560    576.453798    576.244217    578.301376\n",
       "min     -999.983952   -999.968792   -999.000000   -999.000000\n",
       "25%     -502.160012   -500.333306   -492.250000   -498.000000\n",
       "50%        4.057258     -4.377464      3.000000      0.000000\n",
       "75%      502.159214    494.614704    499.000000    500.250000\n",
       "max      999.998666    999.913716    999.000000    999.000000"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data2.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7864347e-60bb-461d-a426-f92728c81ba2",
   "metadata": {},
   "source": [
    "### Performance?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1834f046-0511-4f20-a9e9-3eedf12d6737",
   "metadata": {},
   "source": [
    "You can check which of the two alternatives has the better performance for in-place operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "e4aa3b34-c948-4efe-b523-37e3ab2b2522",
   "metadata": {},
   "outputs": [],
   "source": [
    "nr_rows = 100_000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "018ed5c4-de6f-4a18-bc11-ae3917dcf481",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.DataFrame({\n",
    "    'column1': np.random.uniform(-1_000.0, 1_000.0, size=nr_rows),\n",
    "    'column2': np.random.uniform(-1_000.0, 1_000.0, size=nr_rows),\n",
    "    'column3': np.random.uniform(-1_000, 1_000, size=nr_rows).astype(np.int64),\n",
    "    'column4': np.random.uniform(-1_000, 1_000, size=nr_rows).astype(np.int64),\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "08daae66-9e28-4394-8021-b64cd806a4ab",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6.94 ms ± 638 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "data.clip([-500.0, None, None, None], [500.0, None, None, None], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "04dd1afc-11cb-46a6-9e15-28461c9e96b3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "946 μs ± 49.5 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "data['column1'] = data['column1'].clip(-500.0, 500.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "5bc546df-96b7-446c-be09-3bdc48b6e86b",
   "metadata": {},
   "outputs": [],
   "source": [
    "nr_rows = 10_000_000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "e8d2d195-0c5f-48ee-956f-895c6ae45fa3",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.DataFrame({\n",
    "    'column1': np.random.uniform(-1_000.0, 1_000.0, size=nr_rows),\n",
    "    'column2': np.random.uniform(-1_000.0, 1_000.0, size=nr_rows),\n",
    "    'column3': np.random.uniform(-1_000, 1_000, size=nr_rows).astype(np.int64),\n",
    "    'column4': np.random.uniform(-1_000, 1_000, size=nr_rows).astype(np.int64),\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "5d455f94-47f4-4582-8019-15763457198b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "778 ms ± 78.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "data.clip([-500.0, None, None, None], [500.0, None, None, None], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "b8b68166-7010-41d7-b2ea-f6536dc2d147",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "49 ms ± 6.72 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "data['column1'] = data['column1'].clip(-500.0, 500.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca530907-9cda-408f-816a-f6f34351e0d9",
   "metadata": {},
   "source": [
    "As you can see, assigning the column to perform an in--place operations is significantly faster."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85008d01-6048-43a8-9994-c1ed3e5ffd68",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5561ddaf-f8de-4b1f-a983-07318228e0ae",
   "metadata": {},
   "source": [
    "If your code relies on \"spooky action at a distance\" you have to prepare for the release of pandas 3.x, since your code will no longer work and generate errors.  You can do so by enabling Copy-on-Write via the `options` and fix the resulting issues."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38a0ea3c-3eb1-4dac-9a1d-04d7a366bbbd",
   "metadata": {},
   "source": [
    "In general, it is good practice to enable Copy-on-Write for development of new code as well, since you will be sure that when pandas 3.x is release, your code will at least work without issues related to Copy-on-Write."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf42709d-5376-4af5-9ee4-3e174c601f82",
   "metadata": {},
   "source": [
    "You can find more information and examples in the [pandas documentation](https://pandas.pydata.org/docs/user_guide/copy_on_write.html)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}