notebooks/01_build_pdf_vector_db.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Semantic Search with LangChain, Amazon Titan Embeddings, and FAISS\n",
    "Jupyter notebook for loading documents from PDFs, extracting and splitting text into semantically meaningful chunks using [LangChain](https://python.langchain.com/docs/get_started/introduction), generating text embeddings from those chunks utilizing an , generating embeddings from the text using an  [Amazon Titan Embeddings G1 - Text models](https://docs.aws.amazon.com/bedrock/latest/userguide/titan-embedding-models.html), and storing the embeddings in a [FAISS](https://python.langchain.com/docs/integrations/vectorstores/faiss/) vector database for retrieval.\n",
    "\n",
    "\n",
    "![Diagram](data/build_pdf_vector_db.jpg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requirements: \n",
    "- Install boto3 - This is the [AWS SDK for Python ](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingTheBotoAPI.html)that allows interacting with AWS services. Install with `pip install boto3`.\n",
    "- [Configure AWS credentials](https://docs.aws.amazon.com/braket/latest/developerguide/braket-using-boto3.html) - Boto3 needs credentials to make API calls to AWS.\n",
    "- Install [Langchain](https://python.langchain.com/docs/get_started/introduction), a framework for developing applications powered by large language models (LLMs). Install with `pip install langchain`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install boto3\n",
    "#!pip install langchain\n",
    "#!pip install psycopg2-binary\n",
    "#!pip install langchain_experimental\n",
    "# or install requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'langchain_aws'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[1], line 5\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mtext_splitter\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;28;01mimport\u001b[39;00m RecursiveCharacterTextSplitter \u001b[38;5;66;03m# to split documents into smaller chunks.\u001b[39;00m\n\u001b[1;32m      4\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21;01mlangchain_community\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mvectorstores\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;28;01mimport\u001b[39;00m FAISS \u001b[38;5;66;03m# to store the documents in a vector database.\u001b[39;00m\n\u001b[0;32m----> 5\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21;01mlangchain_aws\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;28;01mimport\u001b[39;00m BedrockEmbeddings \u001b[38;5;66;03m# to create embeddings for the documents.\u001b[39;00m\n\u001b[1;32m      6\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21;01mlangchain_experimental\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mtext_splitter\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;28;01mimport\u001b[39;00m SemanticChunker \u001b[38;5;66;03m# to split documents into smaller chunks.\u001b[39;00m\n\u001b[1;32m      7\u001b[0m \u001b[38;5;66;03m#https://api.python.langchain.com/en/latest/experimental_api_reference.html#module-langchain_experimental.text_splitter\u001b[39;00m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'langchain_aws'"
     ]
    }
   ],
   "source": [
    "import boto3 # to interact with AWS services.\n",
    "from langchain_community.document_loaders import PyPDFLoader, PyPDFDirectoryLoader # to load documents from PDF files.\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter # to split documents into smaller chunks.\n",
    "from langchain_community.vectorstores import FAISS # to store the documents in a vector database.\n",
    "from langchain_aws import BedrockEmbeddings # to create embeddings for the documents.\n",
    "from langchain_experimental.text_splitter import SemanticChunker # to split documents into smaller chunks.\n",
    "#https://api.python.langchain.com/en/latest/experimental_api_reference.html#module-langchain_experimental.text_splitter\n",
    "from langchain_core.documents import Document \n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bedrock_client              = boto3.client(\"bedrock-runtime\", region_name='us-east-1') \n",
    "boto3_bedrock               = boto3.client('bedrock', region_name='us-east-1')\n",
    "bedrock_embeddings          = BedrockEmbeddings(model_id=\"amazon.titan-embed-text-v2:0\",client=bedrock_client)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validate the connection\n",
    "We can check the client works by trying out the list_foundation_models() method, which will tell us all the models available for us to use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[models['modelId'] for models in boto3_bedrock.list_foundation_models()['modelSummaries']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparation: PDF file to VectorDB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`load_and_split_pdf` function load a PDF file, extract the text, and splits it into overlapping chunks based on character offsets using [RecursiveCharacterTextSplitter](https://python.langchain.com/docs/how_to/recursive_text_splitter/).\n",
    "\n",
    "RecursiveCharacterTextSplitter splits a text into smaller chunks based on the maximum number of characters allowed per chunk. It works as follows:\n",
    "\n",
    "1. First, it tries to split the text into chunks using a separator, such as a whitespace or a line break `[\"\\n\\n\", \"\\n\", \" \", \"\"]`.\n",
    "\n",
    "2. If the resulting chunks exceed the maximum character limit, then it recursively splits each chunk into even smaller parts, until no chunk exceeds the limit.\n",
    "\n",
    "3. This recursive process continues until all chunks comply with the maximum character limit.\n",
    "\n",
    "The advantage of using RecursiveCharacterTextSplitter is that it divides the text in a more natural way, respecting sentence and paragraph boundaries whenever possible. This helps preserve the context and meaning of the original text, which is important for NLP tasks such as summarization, text generation, and sentiment analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_name = \"Amazon_Bedrock_User_Guide.pdf\"\n",
    "path_file = \"demo-files\"\n",
    "file_path = f\"{path_file}/{file_name}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_and_split_pdf(chunk_size, chunk_overlap):\n",
    "    text_splitter = RecursiveCharacterTextSplitter(\n",
    "        chunk_size=chunk_size, chunk_overlap=chunk_overlap\n",
    "    )\n",
    "    loader = PyPDFDirectoryLoader(\"demo-files\")\n",
    "    docs = loader.load_and_split(text_splitter)\n",
    "    return docs, loader\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chunk_size = 1000\n",
    "chunk_overlap = 100\n",
    "\n",
    "docs, loader = load_and_split_pdf(chunk_size, chunk_overlap)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"documentos:\", len(docs))\n",
    "docs[6:9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`load_and_split_pdf_semantic` function loads a PDF, splits the text into semantically meaningful chunks using [SemanticChunker](https://python.langchain.com/docs/how_to/semantic-chunker/), and returns the split documents. \n",
    "\n",
    "Unlike [RecursiveCharacterTextSplitter](https://python.langchain.com/docs/how_to/recursive_text_splitter/) which divides the text based on a character limit, [SemanticChunker](https://python.langchain.com/docs/how_to/semantic-chunker/) uses a language model to understand the meaning and context of the text, and then divides it into sections that have a coherent meaning.\n",
    "\n",
    "The process works as follows:\n",
    "\n",
    "1. The full text and a language model are provided to the [SemanticChunker](https://python.langchain.com/docs/how_to/semantic-chunker/).\n",
    "\n",
    "2. The language model analyzes the text and divides it into semantically coherent sentences or segments. \n",
    "\n",
    "3. These segments are grouped into larger \"chunks\" using various techniques, such as topic analysis, topic change detection, etc.\n",
    "\n",
    "4. The resulting \"chunks\" represent sections of the text that have a coherent meaning and context.\n",
    "\n",
    "Additionally, you can use [Breakpoints](https://python.langchain.com/docs/how_to/semantic-chunker/) to have a more granular control over how the text is divided into chunks, which can be important to preserve the meaning and context of the original text during processing.\n",
    "\n",
    "The advantage of using SemanticChunker is that it produces text fragments that are easier to process and understand for subsequent language models, since each fragment has a coherent semantic meaning. This is especially useful for tasks such as summarization, information extraction, and answer generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_and_split_pdf_semantic(file_path, embeddings):\n",
    "    text_splitter = SemanticChunker(embeddings, breakpoint_threshold_amount= 80)\n",
    "    loader = PyPDFLoader(file_path)\n",
    "    docs = loader.load_and_split(text_splitter)\n",
    "    print(f\"docs:{len(docs)}\")\n",
    "    return docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "semantic_docs = load_and_split_pdf_semantic(file_path, bedrock_embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"documentos:\", len(semantic_docs))\n",
    "semantic_docs[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove any empty pages or documents without actual content.\n",
    "clean_docs = []\n",
    "for doc in semantic_docs:\n",
    "    if len(doc.page_content):\n",
    "        clean_docs.append(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(clean_docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for doc in clean_docs:\n",
    "    if len(doc.page_content) == 0:\n",
    "        print(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Build Vector database](https://python.langchain.com/docs/integrations/vectorstores/faiss/#ingestion)\n",
    "Now, using Amazon Bedrock embeddings, create a vector database of document embeddings using [FAISS](https://python.langchain.com/docs/integrations/vectorstores/faiss/) that can allow quick searching by similarity and retrieval of related documents in the future. ,\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "db = FAISS.from_documents(clean_docs, bedrock_embeddings)\n",
    "print(f\"Vector Database:{db.index.ntotal} docs\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create an embedding of the query such that it could be compared with the documents\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"What is a prompt?\"\n",
    "query_embedding = db.embedding_function.embed_query(query)\n",
    "np.array(query_embedding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can query directly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Query directly](https://python.langchain.com/v0.2/api_reference/community/vectorstores/langchain_community.vectorstores.faiss.FAISS.html)\n",
    "Similarity search\n",
    "Performing a simple similarity search with filtering on metadata can be done as follows:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"What is a prompt?\"\n",
    "docs_query = db.similarity_search(query,\n",
    "                                  k=2\n",
    "                                  )\n",
    "print(\"Docs Len: \",len(docs_query))\n",
    "print(docs[0].page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Similarity search with score\n",
    "\n",
    "You can also search with score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = db.similarity_search_with_score(\n",
    "    \"What is a prompt?\", \n",
    "    k=2, \n",
    ")\n",
    "for res, score in results:\n",
    "    print(f\"* [SIM={score:3f}] {res.page_content} [{res.metadata}]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Retriever](https://python.langchain.com/v0.2/api_reference/community/vectorstores/langchain_community.vectorstores.faiss.FAISS.html#langchain_community.vectorstores.faiss.FAISS.as_retriever)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "retriever = db.as_retriever()\n",
    "docs_rt = retriever.invoke(query)\n",
    "print(\"Docs Len: \",len(docs_rt))\n",
    "print(docs[0].page_content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "docs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Save to a local Vector database.](https://python.langchain.com/docs/integrations/vectorstores/faiss/#as-a-retriever)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "db_file_name = file_name.split(\".\")[0]\n",
    "db_file = f\"{db_file_name}.vdb\"\n",
    "db.save_local(db_file)\n",
    "print(f\"vectordb was saved in {db_file}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load and Query local Vector database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "db_file_name = \"Amazon_Bedrock_User_Guide.vdb\"\n",
    "new_db = FAISS.load_local(db_file_name, bedrock_embeddings, allow_dangerous_deserialization=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"What is a prompt?\"\n",
    "docs = new_db.similarity_search(query)\n",
    "print(docs[0].page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Delete Vectordb](https://python.langchain.com/docs/integrations/vectorstores/faiss/#delete)\n",
    "\n",
    "You can also delete records from the vector store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"count before:\", new_db.index.ntotal)\n",
    "new_db.delete([new_db.index_to_docstore_id[0]])\n",
    "print(\"count after:\", new_db.index.ntotal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Delete the entire database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_db.delete"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}