Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tweaks from David and Scott #5

Merged
merged 2 commits into from
Feb 19, 2024
Merged

Tweaks from David and Scott #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f59fa76
Tweaks from David and Scott
epugh Feb 19, 2024
98a0ccc
Tweaks from the team..
epugh Feb 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 10 additions & 19 deletions jupyterlite/files/examples/Fleiss Kappa.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
"cells": [
{
"cell_type": "markdown",
"source": "# Fleiss' Kappa \nTo understand how much your raters what? Scott, need some text!\n\nPlease copy this example and customize it for your own purposes!",
"source": "# Fleiss' Kappa \nTo understand how much your judges agree with each other. It is meant to be used with more than two judges.\n\nRead https://www.datanovia.com/en/blog/kappa-coefficient-interpretation/ to learn more.\n\nPlease copy this example and customize it for your own purposes!",
"metadata": {},
"id": "bd7e4efa-eb00-451e-984d-ed6646d8e25f"
},
Expand Down Expand Up @@ -51,11 +51,11 @@
},
{
"cell_type": "code",
"source": "QUEPID_BOOK_NUM = 25\n\n# Not needed if running within Quepid JupyterLite\n# QUEPID_API_TOKEN = \"\"",
"source": "QUEPID_BOOK_NUM = 25",
"metadata": {
"trusted": true
},
"execution_count": 3,
"execution_count": 2,
"outputs": [],
"id": "71803a49-4065-4adf-a69e-cb0fe2d00f22"
},
Expand All @@ -71,7 +71,7 @@
"metadata": {
"trusted": true
},
"execution_count": 4,
"execution_count": 3,
"outputs": [],
"id": "31193536-98eb-4b46-ab98-af04ee07c6d3"
},
Expand All @@ -81,7 +81,7 @@
"metadata": {
"trusted": true
},
"execution_count": 5,
"execution_count": null,
"outputs": [],
"id": "8fef6231-daa8-467f-ac57-13a144e8a356"
},
Expand All @@ -97,7 +97,7 @@
"metadata": {
"trusted": true
},
"execution_count": 6,
"execution_count": null,
"outputs": [],
"id": "9a8561fd-2dbf-477e-9ac1-4df6d5ebdc91"
},
Expand All @@ -113,7 +113,7 @@
"metadata": {
"trusted": true
},
"execution_count": 7,
"execution_count": null,
"outputs": [],
"id": "a7598308-129b-4628-ad3a-fc3d703f8205"
},
Expand All @@ -129,22 +129,13 @@
"metadata": {
"trusted": true
},
"execution_count": 8,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": "<IPython.core.display.Markdown object>",
"text/markdown": "## Fleiss' Kappa: -0.3333"
},
"metadata": {}
}
],
"execution_count": null,
"outputs": [],
"id": "25a613f9"
},
{
"cell_type": "markdown",
"source": "_This notebook was last updated 17-FEB-2024_",
"source": "_This notebook was last updated 19-FEB-2024_",
"metadata": {},
"id": "5704579e-2321-4629-8de0-6608b428e2b6"
},
Expand Down
69 changes: 62 additions & 7 deletions jupyterlite/files/examples/Jaccard and RBO Comparison.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,12 @@
"cells": [
{
"cell_type": "markdown",
"source": "# Jaccard and RBO Comparison \nTo understand the magnatude of changes to your query result sets, you can compare multiple snapshots together, either from the same case or different cases. \n\nThis notebook provides both Jaccard and Rank Biased Overlap (RBO) metrics.\n\nPlease copy this example and customize it for your own purposes!",
"source": "# Jaccard and RBO Comparison \nTo understand the magnatude of changes to your query result sets, you can compare multiple snapshots to each other.\n\nThis notebook provides both Jaccard and Rank Biased Overlap (RBO) metrics.\n\nPlease copy this example and customize it for your own purposes!",
"metadata": {}
},
{
"cell_type": "code",
"source": "from js import fetch\nfrom typing import List, Optional, Union\n\nimport json\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\n\nimport piplite\nawait piplite.install('seaborn')\nawait piplite.install('rbo')\n\nimport rbo\nimport seaborn as sns\n\nimport os\n\nos.environ[\"TQDM_DISABLE\"] = \"1\"",
"source": "from js import fetch\nfrom typing import List, Optional, Union\n\nimport json\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\n\nimport piplite\nawait piplite.install('seaborn')\nawait piplite.install('rbo')\n\nimport rbo\nimport seaborn as sns\n\nimport os",
"metadata": {
"trusted": true
},
Expand Down Expand Up @@ -72,16 +72,30 @@
}
]
},
{
"cell_type": "code",
"source": "os.environ[\"TQDM_DISABLE\"] = \"1\"",
"metadata": {
"trusted": true
},
"execution_count": 5,
"outputs": []
},
{
"cell_type": "code",
"source": "def jaccard(l1, l2, max_n):\n if len(l1) == 0 and len(l2) == 0:\n return 1\n max_len = min(len(l1), len(l2), max_n)\n set1 = set(l1[:max_len])\n set2 = set(l2[:max_len])\n intersection = len(set1.intersection(set2))\n union = len(set1) + len(set2) - intersection\n return float(intersection) / union\n\nasync def load_snapshots(case_id1, snapshot_id1, case_id2, snapshot_id2):\n df_a = await load_snapshot(case_id1, snapshot_id1)\n df_b = await load_snapshot(case_id2, snapshot_id2)\n return df_a.merge(df_b, on='query')\n\nasync def compare(case_id1, snapshot_id1, case_id2, snapshot_id2):\n df = await load_snapshots(case_id1, snapshot_id1, case_id2, snapshot_id2)\n \n df['jaccard'] = df.apply(lambda row: jaccard(row['docs_x'], row['docs_y'], 10), axis=1)\n df['rbo'] = df.apply(lambda row: rbo.RankingSimilarity(row['docs_x'], row['docs_y']).rbo(), axis=1)\n df['score_delta'] = df['score_y'] - df['score_x']\n df.name = f\"Case {case_id1} snapshot {snapshot_id1} vs. case {case_id1} snapshot {snapshot_id2}\"\n return df\n\n\n\nawait compare(case_id1=6789, snapshot_id1=2471, case_id2=6789, snapshot_id2=2472)",
"metadata": {
"trusted": true
},
"execution_count": 7,
"execution_count": 6,
"outputs": [
{
"execution_count": 7,
"name": "stderr",
"text": "/lib/python3.11/site-packages/rbo/rbo.py:129: TqdmMonitorWarning: tqdm:disabling monitor support (monitor_interval = 0) due to:\ncan't start new thread\n for d in tqdm(range(1, k), disable=~self.verbose):\n",
"output_type": "stream"
},
{
"execution_count": 6,
"output_type": "execute_result",
"data": {
"text/plain": " num_results_x score_x \\\nquery \nprojector screen 1 1.0 \nnotebook 1 1.0 \niphone 8 1 1.0 \nprinter 1 1.0 \ncomputer 1 1.0 \n... ... ... \nwindows 10 1 1.0 \nmicrowave 1 1.0 \nbluetooth speakers 1 1.0 \ncoffee 1 1.0 \nvans 1 1.0 \n\n docs_x \\\nquery \nprojector screen [1069226, 47471, 490523, 1229109, 1229118, 325... \nnotebook [3851056, 3959000, 1550833, 1684763, 1675257, ... \niphone 8 [2048598, 1648546, 79524888, 1857711, 3613408,... \nprinter [3849563, 2225354, 1569761, 798960, 377837, 13... \ncomputer [560468, 532095, 560475, 523407, 693956, 56047... \n... ... \nwindows 10 [4481689, 3902727, 1560529, 1797902, 3155116, ... \nmicrowave [79513345, 4020048, 1768856, 2936032] \nbluetooth speakers [1993197, 3537784, 279672, 2663204, 558184, 33... \ncoffee [1996660, 2102472, 79583150, 1357989, 656359, ... \nvans [78503576, 79118095, 77388459, 78322005, 79013... \n\n num_results_y score_y \\\nquery \nprojector screen 1 1.0 \nnotebook 1 1.0 \niphone 8 1 1.0 \nprinter 1 1.0 \ncomputer 1 1.0 \n... ... ... \nwindows 10 1 1.0 \nmicrowave 1 1.0 \nbluetooth speakers 1 1.0 \ncoffee 1 1.0 \nvans 1 1.0 \n\n docs_y \\\nquery \nprojector screen [1069226, 47471, 490523, 1229109, 1229118, 325... \nnotebook [3851056, 3959000, 1550833, 1684763, 1675257, ... \niphone 8 [2048598, 1648546, 79524888, 1857711, 3613408,... \nprinter [3849563, 2225354, 1569761, 798960, 377837, 13... \ncomputer [560468, 532095, 560475, 523407, 693956, 56047... \n... ... \nwindows 10 [4481689, 3902727, 1560529, 1797902, 3155116, ... \nmicrowave [79513345, 4020048, 1768856, 2936032] \nbluetooth speakers [1993197, 3537784, 279672, 2663204, 558184, 33... \ncoffee [1996660, 2102472, 79583150, 1357989, 656359, ... \nvans [78503576, 79118095, 77388459, 78322005, 79013... \n\n jaccard rbo score_delta \nquery \nprojector screen 1.0 1.0 0.0 \nnotebook 1.0 1.0 0.0 \niphone 8 1.0 1.0 0.0 \nprinter 1.0 1.0 0.0 \ncomputer 1.0 1.0 0.0 \n... ... ... ... \nwindows 10 1.0 1.0 0.0 \nmicrowave 1.0 1.0 0.0 \nbluetooth speakers 1.0 1.0 0.0 \ncoffee 1.0 1.0 0.0 \nvans 1.0 1.0 0.0 \n\n[135 rows x 9 columns]",
Expand All @@ -93,11 +107,45 @@
},
{
"cell_type": "code",
"source": "import matplotlib\nmatplotlib.rc_file_defaults()\n\ndef plot_compare(df):\n figure, axes = plt.subplots(1, 3, figsize=(10, 4))\n figure.suptitle(df.name)\n\n sns.barplot(ax=axes[0], x=df['score_delta'], y=df.index, width=0.3, color='darkgrey')\n axes[0].set(xlim=(-1, 1))\n axes[0].set_xlabel('Change in Score')\n axes[0].set_ylabel('')\n axes[0].set_facecolor((0.90, 0.90, 0.90))\n axes[0].grid(True)\n axes[0].spines['top'].set_visible(False)\n axes[0].spines['right'].set_visible(False)\n axes[0].spines['bottom'].set_visible(False)\n axes[0].spines['left'].set_visible(False)\n axes[0].set_axisbelow(True)\n axes[0].xaxis.grid(color='w', linestyle='solid')\n axes[0].yaxis.grid(color='w', linestyle='solid')\n \n sns.heatmap(df[['jaccard']], ax=axes[1], cmap='crest', annot=True, xticklabels=False, yticklabels=False)\n axes[1].set_xlabel('Jaccard Similiarity')\n axes[1].set_ylabel('')\n \n sns.heatmap(df[['rbo']], ax=axes[2], cmap='crest', annot=True, xticklabels=False, yticklabels=False)\n axes[2].set_xlabel('Rank Biased Overlap')\n axes[2].set_ylabel('')\n \n plt.show()\n \ndf = await compare(case_id1=6789, snapshot_id1=2471, case_id2=6789, snapshot_id2=2473)\nplot_compare(df)",
"source": "import matplotlib\nmatplotlib.rc_file_defaults()\n\ndef plot_compare(df):\n figure, axes = plt.subplots(1, 3, figsize=(10, 4))\n figure.suptitle(df.name)\n\n sns.barplot(ax=axes[0], x=df['score_delta'], y=df.index, width=0.3, color='darkgrey')\n axes[0].set(xlim=(-1, 1))\n axes[0].set_xlabel('Change in Score')\n axes[0].set_ylabel('')\n axes[0].set_facecolor((0.90, 0.90, 0.90))\n axes[0].grid(True)\n axes[0].spines['top'].set_visible(False)\n axes[0].spines['right'].set_visible(False)\n axes[0].spines['bottom'].set_visible(False)\n axes[0].spines['left'].set_visible(False)\n axes[0].set_axisbelow(True)\n axes[0].xaxis.grid(color='w', linestyle='solid')\n axes[0].yaxis.grid(color='w', linestyle='solid')\n \n sns.heatmap(df[['jaccard']], ax=axes[1], cmap='crest', annot=True, xticklabels=False, yticklabels=False)\n axes[1].set_xlabel('Jaccard Similiarity')\n axes[1].set_ylabel('')\n \n sns.heatmap(df[['rbo']], ax=axes[2], cmap='crest', annot=True, xticklabels=False, yticklabels=False)\n axes[2].set_xlabel('Rank Biased Overlap')\n axes[2].set_ylabel('')\n \n plt.show()\n \ndf = await compare(case_id1=6789, snapshot_id1=2471, case_id2=6789, snapshot_id2=2473)\n",
"metadata": {
"trusted": true
},
"execution_count": 6,
"execution_count": 7,
"outputs": []
},
{
"cell_type": "markdown",
"source": "## Overall Jaccard and RBO Scores",
"metadata": {}
},
{
"cell_type": "code",
"source": "print(f\"Overall Jaccard Score: {df['jaccard'].mean()}\\nOverall RBO Score: {df['rbo'].mean()}\")",
"metadata": {
"trusted": true
},
"execution_count": 8,
"outputs": [
{
"name": "stdout",
"text": "Overall Jaccard Score: 1.0\nOverall RBO Score: 1.0\n",
"output_type": "stream"
}
]
},
{
"cell_type": "markdown",
"source": "## Query Level Jaccard and RBO Scores",
"metadata": {}
},
{
"cell_type": "code",
"source": "plot_compare(df)",
"metadata": {
"trusted": true
},
"execution_count": 9,
"outputs": [
{
"output_type": "display_data",
Expand All @@ -111,8 +159,15 @@
},
{
"cell_type": "markdown",
"source": "_This notebook was last updated 16-FEB-2024_",
"source": "_This notebook was last updated 19-FEB-2024_",
"metadata": {}
},
{
"cell_type": "code",
"source": "",
"metadata": {},
"execution_count": null,
"outputs": []
}
]
}
Loading
Loading