01_standalone_examples/lakefs-mount-demo/lakeFS Hugging Face Mount Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0c053f0c-88da-4972-bdbe-686a37af7325",
   "metadata": {},
   "source": [
    "# Fast Data Loading and Reproducibility of Hugging Face Datasets for Deep Learning Workloads with lakeFS Mount\n",
    "\n",
    "Use Case: Mount lakeFS datasets on laptop or server with/without GPUs for AI/ML use cases\n",
    "\n",
    "Watch [this video](https://www.youtube.com/watch?v=BgKuoa8LAaU) to understand the use case as well as the demo.\n",
    "\n",
    "[Contact lakeFS](https://lakefs.io/contact-sales/) to get the lakeFS Everest binary for Linux x86_64 OS. Download and save the binary on your laptop inside \"lakeFS-samples/01_standalone_examples/lakefs-mount-demo\" folder."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fea8ed1f-74f0-40fe-aa8f-f4548a108c28",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Config"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd4b5229-ed90-4ff0-893b-dcfdddec161f",
   "metadata": {},
   "source": [
    "### lakeFS endpoint and credentials\n",
    "\n",
    "Change these if using lakeFS other than provided in the samples repo. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e626704f-1479-47ad-8638-ea010c1432d5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "lakefsEndPoint = 'http://lakefs:8000' # e.g. 'https://username.aws_region_name.lakefscloud.io' \n",
    "lakefsAccessKey = 'AKIAIOSFOLKFSSAMPLES'\n",
    "lakefsSecretKey = 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7c151e3-c469-4258-a7e3-9d25c00a9cc5",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Storage Information\n",
    "\n",
    "If you're not using sample repo lakeFS, then change the Storage Namespace to a location in the bucket you’ve configured. The storage namespace is a location in the underlying storage where data for this repository will be stored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5c64b30-4e57-40bd-aa76-fdaf6000f7c0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "storageNamespace = 's3://example' # e.g. \"s3://bucket\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb9e61a4-052c-4992-92c4-103fd68552ac",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Setup\n",
    "\n",
    "**(you shouldn't need to change anything in this section, just run it)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6213e05b-03d4-4065-b92d-b189eec16206",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "repo_name = \"lakefs-mount-hugging-face-demo\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e898140-531c-45ce-964b-47bbc56718f2",
   "metadata": {},
   "source": [
    "### Versioning Information "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4472c870-8034-47da-9f7a-7b814b77f63e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sourceBranch = \"main\"\n",
    "experimentBranch = \"experiment\"\n",
    "no_of_experiments = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f18f02c-03fd-4781-a274-760627fe9c27",
   "metadata": {},
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a5c74d0-5c09-4a27-af1d-a37b1981eada",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import lakefs\n",
    "from assets.lakefs_demo import print_commit\n",
    "from datasets import load_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71578731-133a-4af8-a6d7-db1f29f12de3",
   "metadata": {},
   "source": [
    "### Set environment variables and create lakectl.yaml file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "209195d6-7335-4c8f-8ea1-e1cad09e4310",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "os.environ[\"LAKECTL_SERVER_ENDPOINT_URL\"] = lakefsEndPoint\n",
    "os.environ[\"LAKECTL_CREDENTIALS_ACCESS_KEY_ID\"] = lakefsAccessKey\n",
    "os.environ[\"LAKECTL_CREDENTIALS_SECRET_ACCESS_KEY\"] = lakefsSecretKey"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2065ffa6-73b0-4260-bf7d-ab06f84b9b52",
   "metadata": {},
   "outputs": [],
   "source": [
    "lakectl_file_content = f\"server:\\n    endpoint_url: {lakefsEndPoint}\\ncredentials:\\n    access_key_id: {lakefsAccessKey}\\n    secret_access_key: {lakefsSecretKey}\"\n",
    "! echo -e \"$lakectl_file_content\" > .lakectl.yaml\n",
    "! cat .lakectl.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0cd1b6e-e080-425f-a99b-d85212ae0a44",
   "metadata": {},
   "source": [
    "### Verify lakeFS credentials by getting lakeFS version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e7e745c-a001-4ef9-aab3-53639174e5e5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(\"Verifying lakeFS credentials…\")\n",
    "try:\n",
    "    v=lakefs.client.Client().version\n",
    "except:\n",
    "    print(\"🛑 failed to get lakeFS version\")\n",
    "else:\n",
    "    print(f\"…✅lakeFS credentials verified\\n\\nℹ️lakeFS version {v}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07041af8-fae2-4064-94c5-afc758695903",
   "metadata": {},
   "source": [
    "### Define lakeFS Repository"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca4cf7b6-b01b-4d79-bca1-4d30bd3ba7e8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "repo = lakefs.Repository(repo_name).create(storage_namespace=f\"{storageNamespace}/{repo_name}\", default_branch=sourceBranch, exist_ok=True)\n",
    "branchMain = repo.branch(sourceBranch)\n",
    "print(repo)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a53d3d3-c99f-4ddd-9486-cf6f3720086f",
   "metadata": {},
   "source": [
    "# Main demo starts here 🚦 👇🏻"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9df6a0e3-c8a2-4a35-9ff9-150220dc508f",
   "metadata": {},
   "source": [
    "### Create an empty Git repository and configure Git. Git will version control your code (Python programs in this example) while lakeFS will version control your data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f77c01d-3632-4962-83f2-1aa077fe9b81",
   "metadata": {},
   "outputs": [],
   "source": [
    "! git init {repo_name}\n",
    "! git config --global user.email \"you@example.com\"\n",
    "! git config --global user.name \"Your Name\"\n",
    "! cd {repo_name} && git checkout -b main\n",
    "! cp -t {repo_name} 'ReadDataset.py' 'Preprocess.py'\n",
    "! cd {repo_name} && git add -A && git status && git commit -m \"Added code\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "051af1c6-be68-416b-888d-b88766a5d966",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Load Hugging Face dataset and save it to lakeFS repository"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca09dcc0-9630-4f1b-b4c9-2c7ed034e85b",
   "metadata": {},
   "outputs": [],
   "source": [
    "hugging_face_dataset_name = \"beans\"\n",
    "lakefs_path_for_dataset = f'lakefs://{repo_name}/{sourceBranch}/datasets'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1076ab00-fcca-464f-8b04-ebe52af287e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = load_dataset(hugging_face_dataset_name, split=\"train\")\n",
    "dataset.save_to_disk(f'{lakefs_path_for_dataset}/{hugging_face_dataset_name}/')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f11730f",
   "metadata": {},
   "source": [
    "### Commit changes and attach some metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6dc6db1-11cb-41f6-93fc-432e6cc97386",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "kwargs={'allow_empty': True}\n",
    "ref = branchMain.commit(message='Uploaded Hugging Face dataset!', metadata={'using': 'python_sdk'}, **kwargs )\n",
    "print_commit(ref.get_commit())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff3443cb-f983-423a-b14a-a79b5ae95972",
   "metadata": {},
   "source": [
    "### Create multiple branches to run multiple experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6daf9d9-fa5a-418d-b4c0-5641debc6e5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "for N in range(1, no_of_experiments+1):\n",
    "    experimentBranchN = f'{experimentBranch}-{N}'\n",
    "    branchExperiment = repo.branch(f'{experimentBranchN}').create(source_reference=sourceBranch, exist_ok=True)\n",
    "    print(f\"{experimentBranchN} ref:\", branchExperiment.get_commit().id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1568d4f-a86d-4124-8c59-e54fc409d79a",
   "metadata": {},
   "source": [
    "### Create Git branches and mount lakeFS data path as local filesystem for multiple experiments.\n",
    "#### The \"git add\" command adds changes in the working directory to the staging area.\n",
    "#### Git doesn't add data to staging area while adds \".everest/source\" file which includes lakeFS mount path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df03bab0-5df3-47b6-b016-7acce15ee40b",
   "metadata": {},
   "outputs": [],
   "source": [
    "for N in range(1, no_of_experiments+1):\n",
    "    experimentBranchN = f'{experimentBranch}-{N}'\n",
    "    print(f'Mount {experimentBranchN} dataset')\n",
    "    lakefs_path_for_dataset = f'lakefs://{repo_name}/{experimentBranchN}/datasets'\n",
    "    mount_location = f'{experimentBranchN}/datasets'\n",
    "    mount_command = f'../everest mount {lakefs_path_for_dataset} {mount_location} --presign=false --protocol fuse'\n",
    "    system_output = %system cd {repo_name} && git checkout -b $experimentBranchN main && $mount_command| tail -n 1\n",
    "    print(f'{system_output}\\nCommit lakeFS data source file for {experimentBranchN} to Git')\n",
    "    ! cd {repo_name} && git add -A && git commit -m \"Added data source for lakeFS\"\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d822e14-b96c-4ea8-b847-9d63ce728291",
   "metadata": {},
   "source": [
    "### Let's review \".gitignore\" and \".everest/source\" files created by previous Mount command.\n",
    "#### You will notice in .gitignore file that Git will not commit any files in the \"datasets\" folder but will commit \".everest/source\" file which includes lakeFS mount path along with lakeFS commit id. This way code as well as commit information about data will be kept together in the Git repo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13f89254-a0cb-4dba-a9a0-c4e4ae64e968",
   "metadata": {},
   "outputs": [],
   "source": [
    "! cat {repo_name}/{experimentBranch}-1/datasets/.gitignore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "706da0cf-69df-48ea-917b-3dd9fd1525dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "! cat {repo_name}/{experimentBranch}-1/datasets/.everest/source"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a039b8f-bdbd-4fa8-94bf-4dfd69ba07b4",
   "metadata": {},
   "source": [
    "### Data stored in lakeFS can be accessed as regular files locally. All experiments point to the same dataset so far."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd1c2856-5b29-486b-857f-7eef8928a78f",
   "metadata": {},
   "outputs": [],
   "source": [
    "for N in range(1, no_of_experiments+1):\n",
    "    experimentBranchN = f'{experimentBranch}-{N}'\n",
    "    print(f'{experimentBranchN} dataset files')\n",
    "    dataset_location = f'{repo_name}/{experimentBranchN}/datasets/{hugging_face_dataset_name}'\n",
    "    ! ls -lh $dataset_location\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b75849e-f7b9-4e78-9791-8abfad61ed70",
   "metadata": {},
   "source": [
    "### Read the dataset as local dataset using Python.\n",
    "##### You can review [ReadDataset.py](./ReadDataset.py) Python program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9933064-6256-479b-8254-d272ef8d8308",
   "metadata": {},
   "outputs": [],
   "source": [
    "for N in range(1, no_of_experiments+1):\n",
    "    experimentBranchN = f'{experimentBranch}-{N}'\n",
    "    print(f'Read {experimentBranchN} dataset')\n",
    "    mount_location = f'{repo_name}/{experimentBranchN}/datasets'\n",
    "    ! python ReadDataset.py --mount_location $mount_location --dataset_name $hugging_face_dataset_name\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b38041f-b78d-4cc0-adef-61bce88dfea1",
   "metadata": {},
   "source": [
    "### Read and preprocess the dataset using Python.\n",
    "#### Going to use different number of images for different experiments.\n",
    "#### Save subset of data in the lakeFS repository for the reproducibility purpose.\n",
    "##### You can review [Preprocess.py](./Preprocess.py) Python program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "411a4c8c-c47e-4795-a395-c14ecd9322c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "for N in range(1, no_of_experiments+1):\n",
    "    number_of_images = N * 10\n",
    "    experimentBranchN = f'{experimentBranch}-{N}'\n",
    "    mount_location = f'{repo_name}/{experimentBranchN}/datasets'\n",
    "    print(f'Preprocess {experimentBranchN} dataset to select {number_of_images} images')\n",
    "    ! python Preprocess.py --number_of_images $number_of_images --repo_name $repo_name --branch_name $experimentBranchN --mount_location $mount_location --dataset_name $hugging_face_dataset_name\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5e8342b-2a89-437b-8bce-16addc26e7d6",
   "metadata": {},
   "source": [
    "## Reproducibility use case"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a9f43de-7b09-4540-b867-7c4401a158e8",
   "metadata": {},
   "source": [
    "### You can clone the Git repo in future to reproduce the code as well as data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e68b7bb-a2a7-4fe2-b7de-20c2c99e223c",
   "metadata": {},
   "outputs": [],
   "source": [
    "! git clone ./{repo_name} reproduce"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df43ac93-2ac3-4327-a0c9-7112af78b515",
   "metadata": {},
   "source": [
    "### Mount dataset for previous experiments for reproducibility purpose"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15d4cee5-7fe6-47a5-9f4e-32c208588751",
   "metadata": {},
   "outputs": [],
   "source": [
    "for N in range(1, no_of_experiments+1):\n",
    "    experimentBranchN = f'{experimentBranch}-{N}'\n",
    "    print(f'Mount {experimentBranchN} dataset')\n",
    "    lakefs_path_for_dataset = f'lakefs://{repo_name}/{experimentBranchN}/datasets'\n",
    "    mount_location = f'reproduce/{experimentBranchN}/datasets'\n",
    "    mount_command = f'./everest mount {lakefs_path_for_dataset} {mount_location} --presign=false --protocol fuse'\n",
    "    ! rm -r reproduce/$experimentBranchN/datasets/.everest\n",
    "    system_output = %system $mount_command| tail -n 1\n",
    "    print(f\"{system_output}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1451c560-9eb6-41c0-86bc-fed3d7a15255",
   "metadata": {},
   "source": [
    "### List datasets for different experiments\n",
    "##### You will notice different file size for different experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea5f21b6-ff23-4cc4-ad5a-84d79f6dd2b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "for N in range(1, no_of_experiments+1):\n",
    "    experimentBranchN = f'{experimentBranch}-{N}'\n",
    "    print(f'{experimentBranchN} dataset files')\n",
    "    mount_location = f'reproduce/{experimentBranchN}/datasets/{hugging_face_dataset_name}_subset'\n",
    "    ! ls -lh $mount_location\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0ab9822-68ba-4834-b610-58c89b5c6651",
   "metadata": {},
   "source": [
    "### Read the dataset from previous experiments using Python\n",
    "##### You will notice that each experiment used different number of images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4129ee4-693f-459e-ad0d-62ca46ae637d",
   "metadata": {},
   "outputs": [],
   "source": [
    "for N in range(1, no_of_experiments+1):\n",
    "    experimentBranchN = f'{experimentBranch}-{N}'\n",
    "    print(f'Read {experimentBranchN} dataset')\n",
    "    mount_location = f'reproduce/{experimentBranchN}/datasets'\n",
    "    hugging_face_dataset_name_subset = f'{hugging_face_dataset_name}_subset'\n",
    "    ! python ReadDataset.py --mount_location $mount_location --dataset_name $hugging_face_dataset_name_subset\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cfc1779-7777-4059-93c1-fe40cbc3c0b6",
   "metadata": {},
   "source": [
    "# Demo ends"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b853b30f-d87f-4844-99d7-e105556c7452",
   "metadata": {},
   "source": [
    "## Demo cleanup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "206ade81-6cd8-4417-8141-ff33021cbc24",
   "metadata": {},
   "source": [
    "### Unmount datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35dbbd7b-68b1-4c5e-9f88-7d611174bf86",
   "metadata": {},
   "outputs": [],
   "source": [
    "for N in range(1, no_of_experiments+1):\n",
    "    experimentBranchN = f'{experimentBranch}-{N}'\n",
    "    print(f'Unmount {experimentBranchN} dataset')\n",
    "    mount_location = f'{repo_name}/{experimentBranchN}/datasets'\n",
    "    ! ./everest unmount {mount_location}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fab0686-e9cd-42dc-b418-47da51dcacaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "for N in range(1, no_of_experiments+1):\n",
    "    experimentBranchN = f'{experimentBranch}-{N}'\n",
    "    print(f'Unmount reproduced {experimentBranchN} dataset')\n",
    "    mount_location = f'reproduce/{experimentBranchN}/datasets'\n",
    "    ! ./everest unmount {mount_location}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84750df3-b1aa-4edf-89b6-b80f048bc7bb",
   "metadata": {},
   "source": [
    "### Delete local Git repos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f3f2943-deaf-455b-8cd6-c5aee7043cfb",
   "metadata": {},
   "outputs": [],
   "source": [
    "! rm -r $repo_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bb7bda7-ad67-4242-aa4c-42f33f9f3eb2",
   "metadata": {},
   "outputs": [],
   "source": [
    "! rm -r reproduce"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "135738c2-2f58-4d24-b474-323dd8415b08",
   "metadata": {},
   "source": [
    "### Delete lakeFS branches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a71d2a5-4977-45b4-8a74-bbedadc1a0e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "for N in range(1, no_of_experiments+1):\n",
    "    experimentBranchN = f'{experimentBranch}-{N}'\n",
    "    print(f'Delete {experimentBranchN} branch')\n",
    "    repo.branch(f'{experimentBranchN}').delete();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "873b7142",
   "metadata": {},
   "source": [
    "## More Questions?\n",
    "\n",
    "###### Join the lakeFS Slack group - https://lakefs.io/slack"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}