LlamaIndex_RAG_SelfChunking.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RAG Chunking Analysis - Self Chunking\n",
    "\n",
    "Investigating the chunking of the documents and its effect on getting good results.\n",
    "\n",
    "This notebook is an initial review of the chunking mechanism and creating manual chunks that align better with the document text - in this case making sure numbered and bullet pointed lists are kept with their preceding paragraph.\n",
    "\n",
    "Additionally, it looks at token length of the chunks and uses that to determine how many chunks to retrieve to pass to the LLM for context. This helps to give as much as context as we can to the LLM.\n",
    "\n",
    "This notebook uses the Mistral 7B Instruct LLM. Others will be tried separately."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start with a short story created by ChatGPT and stored in 3 Microsoft Word documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read the documents\n",
    "\n",
    "from llama_index.core import SimpleDirectoryReader\n",
    "\n",
    "documents = SimpleDirectoryReader(\"./Data/\").load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(id_='b7ba673a-8e24-4aaa-9e6d-fcce22f8efb4', embedding=None, metadata={'file_name': 'Thundertooth Part 1.docx', 'file_path': 'Data/Thundertooth Part 1.docx', 'file_type': 'application/vnd.openxmlformats-officedocument.wordprocessingml.document', 'file_size': 14575, 'creation_date': '2024-02-22', 'last_modified_date': '2024-02-22', 'last_accessed_date': '2024-02-22'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={}, text='Thundertooth\\n\\n\\n\\nOnce upon a time, in a prehistoric land filled with dense forests and roaring rivers, there lived a dinosaur named Thundertooth. Thundertooth was no ordinary dinosaur; he possessed the rare ability to speak, a talent that set him apart from his ancient companions. One fateful day, as Thundertooth was basking in the warmth of the sun, a mysterious portal opened before him, and he found himself hurtling through time and space.\\n\\n\\n\\nAs the dazzling vortex subsided, Thundertooth opened his eyes to a world unlike anything he had ever seen. The air was filled with the hum of engines, and towering structures reached towards the sky. Thundertooth\\'s surroundings were a blend of metal and glass, and he quickly realized that he had been transported to a future era.\\n\\n\\n\\nThe once mighty dinosaur now stood bewildered in the midst of a bustling city. Above him, sleek flying cars zipped through the air, leaving trails of neon lights in their wake. Thundertooth felt like an ancient relic in this technological jungle, lost and out of place. With each step, he marveled at the skyscrapers that loomed overhead, their surfaces reflecting the myriad lights of the city.\\n\\n\\n\\nHowever, as night fell, Thundertooth\\'s stomach growled loudly. He realized that he was hungry, and the once vibrant city now seemed like a daunting maze of unfamiliar smells and sights. He wandered through the streets, his massive form drawing astonished stares from the futuristic inhabitants.\\n\\n\\n\\nThundertooth faced a dilemma – he was hungry, but he couldn\\'t bring himself to feast on the humans who scurried around like ants. As his hunger grew, he stumbled upon a park, an oasis of greenery amidst the concrete and steel. The park was adorned with holographic flowers that emitted a gentle glow, creating an ethereal atmosphere.\\n\\n\\n\\nWhile Thundertooth marveled at the beauty of the park, the mayor of the city happened to be passing by. Mayor Eleanor Grace, a charismatic and forward-thinking leader, was immediately intrigued by the sight of the talking dinosaur. She approached Thundertooth with a mix of curiosity and caution.\\n\\n\\n\\n\"Hello there, majestic creature. What brings you to our time?\" Mayor Grace inquired, her voice calm and reassuring.\\n\\n\\n\\nThundertooth, though initially startled, found comfort in the mayor\\'s soothing tone. In broken sentences, he explained his journey through time, the strange portal, and his hunger dilemma. Mayor Grace listened intently, her eyes widening with amazement at the tale of the prehistoric dinosaur navigating the future.\\n\\n\\n\\nRealizing the dinosaur\\'s predicament, Mayor Grace extended an invitation. \"You are welcome in our city, Thundertooth. We can find a way to provide for you without causing harm to anyone. Let us work together to find a solution.\"\\n\\n\\n\\nGrateful for the mayor\\'s hospitality, Thundertooth followed her through the city. Together, they explored the futuristic marketplaces and innovative food labs, eventually discovering a sustainable solution that satisfied the dinosaur\\'s hunger without compromising the well-being of the city\\'s inhabitants.\\n\\n\\n\\nAs the news of Thundertooth\\'s arrival spread, the city embraced the talking dinosaur as a symbol of unity between the past and the future. Thundertooth found a new home in the city\\'s park, where holographic flowers bloomed, and the citizens marveled at the beauty of coexistence across time. And so, in this extraordinary city of flying cars and advanced technology, Thundertooth became a beloved figure, a living bridge between eras, teaching the people that understanding and cooperation could overcome even the greatest challenges.', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " Document(id_='1c73be94-b70d-4ea9-8b0c-993875e7fb94', embedding=None, metadata={'file_name': 'Thundertooth Part 2.docx', 'file_path': 'Data/Thundertooth Part 2.docx', 'file_type': 'application/vnd.openxmlformats-officedocument.wordprocessingml.document', 'file_size': 14334, 'creation_date': '2024-02-22', 'last_modified_date': '2024-02-22', 'last_accessed_date': '2024-02-22'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={}, text=\"Thundertooth\\n\\n\\n\\nEmbraced by the futuristic city and its inhabitants, Thundertooth found a sense of purpose beyond merely satisfying his hunger. Inspired by the advanced technology surrounding him, he decided to channel his creativity into something extraordinary. With the help of the city's brilliant engineers, Thundertooth founded a one-of-a-kind toy factory that produced amazing widgets – magical, interactive toys that captivated the hearts of both children and adults alike.\\n\\n\\n\\nThundertooth's toy factory became a sensation, and its creations were highly sought after. The widgets incorporated cutting-edge holographic displays, levitation technology, and even the ability to change shapes and colors with a mere thought. Children across the city rejoiced as they played with these incredible toys that seemed to bring their wildest fantasies to life.\\n\\n\\n\\nAs the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\\n\\n\\n\\nLumina: The eldest of Thundertooth's children, Lumina inherited her mother's intelligence and her father's sense of wonder. With sparkling scales that emitted a soft glow, Lumina had the ability to generate light at will. She became fascinated with technology, often spending hours tinkering with gadgets and inventing new ways to enhance the widgets produced in the family's factory.\\n\\n\\n\\nEcho: The second-born, Echo, had a gift for mimicry. He could perfectly replicate any sound or voice he heard, providing entertainment to the entire city. His playful nature and ability to bring joy to those around him made him a favorite among the neighborhood children.\\n\\n\\n\\nSapphire: Sapphire, the third sibling, had scales that shimmered like precious gems. She possessed a unique talent for calming and healing, a trait she inherited from both her parents. Whenever someone in the city felt stressed or unwell, Sapphire would extend her gentle touch, bringing comfort and tranquility.\\n\\n\\n\\nIgnis: The youngest of the family, Ignis, had fiery red scales that hinted at his exceptional ability – the power to control small flames. While initially seen as a potential hazard, Ignis channeled his fiery talents into creating mesmerizing light shows, becoming the city's favorite entertainer during festivals and celebrations.\\n\\n\\n\\nThundertooth and Seraphina reveled in the joy of parenthood, watching their children grow and flourish in the futuristic landscape they now called home. The family became an integral part of the city's fabric, not only through the widgets produced in their factory but also through the positive impact each member had on the community.\\n\\n\\n\\nThe toy factory became a symbol of innovation and unity, bringing together dinosaurs and humans in a shared appreciation for creativity and wonder. Thundertooth's legacy extended beyond his time-traveling adventure, leaving an indelible mark on the city and its inhabitants, reminding them that sometimes, the most magical things could emerge from the most unexpected places.\", start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " Document(id_='fd3859d2-0a50-41c7-a4d5-426e87a24d7c', embedding=None, metadata={'file_name': 'Thundertooth Part 3.docx', 'file_path': 'Data/Thundertooth Part 3.docx', 'file_type': 'application/vnd.openxmlformats-officedocument.wordprocessingml.document', 'file_size': 14597, 'creation_date': '2024-02-22', 'last_modified_date': '2024-02-22', 'last_accessed_date': '2024-02-22'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={}, text='Thundertooth\\n\\n\\n\\nOne fateful day, as the citizens of the futuristic city went about their daily lives, a collective gasp echoed through the streets as a massive meteor hurtled towards Earth. Panic spread like wildfire as people looked to the sky in horror, realizing the impending catastrophe. The city\\'s advanced technology detected the threat, and an emergency broadcast echoed through the streets, urging everyone to seek shelter.\\n\\n\\n\\nThundertooth, ever the protector of his newfound home, wasted no time. With a determined gleam in his eyes, he gathered his family and hurried to the city\\'s command center, where Mayor Grace and the leading scientists were coordinating the evacuation efforts.\\n\\n\\n\\nThe mayor, recognizing Thundertooth\\'s intelligence and resourcefulness, approached him. \"Thundertooth, we need a plan to divert or neutralize the meteor. Our technology can only do so much, but with your unique abilities, perhaps we can find a solution.\"\\n\\n\\n\\nThundertooth nodded, understanding the gravity of the situation. He gathered Lumina, Echo, Sapphire, and Ignis, explaining the urgency and the role each of them would play in the impending crisis.\\n\\n\\n\\n1. **Lumina**: Utilizing her deep understanding of technology, Lumina would enhance the city\\'s energy systems to generate a powerful force field, providing a protective barrier against the meteor\\'s impact.\\n\\n\\n\\n2. **Echo**: With his extraordinary mimicry abilities, Echo would amplify the emergency signals, ensuring that every citizen received timely warnings and instructions for evacuation.\\n\\n\\n\\n3. **Sapphire**: Harnessing her calming and healing powers, Sapphire would assist in calming the panicked masses, ensuring an orderly and efficient evacuation.\\n\\n\\n\\n4. **Ignis**: Drawing upon his fiery talents, Ignis would create controlled bursts of heat, attempting to alter the meteor\\'s trajectory and reduce its destructive force.\\n\\n\\n\\nAs the citizens evacuated to designated shelters, the Thundertooth family sprang into action. Lumina worked tirelessly to strengthen the city\\'s energy systems, Echo echoed evacuation orders through the city\\'s speakers, Sapphire offered comfort to those in distress, and Ignis unleashed controlled bursts of flames towards the approaching meteor.\\n\\n\\n\\nThundertooth stood at the forefront, using his mighty roar to coordinate and inspire the efforts of the city\\'s inhabitants. The ground trembled as the meteor drew closer, but the Thundertooth family\\'s coordinated efforts began to take effect. Lumina\\'s force field shimmered to life, deflecting the meteor\\'s deadly path. Echo\\'s amplified warnings reached every corner of the city, ensuring that no one was left behind.\\n\\n\\n\\nAs Ignis\\'s controlled bursts of flames interacted with the meteor, it began to change course. The combined efforts of the Thundertooth family, guided by their unique talents, diverted the catastrophic collision. The meteor, once destined for destruction, now harmlessly sailed past the Earth, leaving the city and its inhabitants unscathed.\\n\\n\\n\\nThe citizens, emerging from their shelters, erupted into cheers of gratitude. Mayor Grace approached Thundertooth, expressing her heartfelt thanks for the family\\'s heroic efforts. The Thundertooth family, tired but triumphant, basked in the relief of having saved their beloved city from imminent disaster.\\n\\n\\n\\nIn the wake of the crisis, the citizens of the futuristic city hailed Thundertooth and his family as true heroes. The toy factory that once brought joy to children now became a symbol of resilience and unity. The Thundertooth family\\'s legacy was forever etched in the city\\'s history, a testament to the power of cooperation and the extraordinary capabilities that could emerge when dinosaurs and humans worked together for the greater good.', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n')]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the GTE Large embeddings from HuggingFace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "config.json: 100%|██████████| 619/619 [00:00<00:00, 3.43MB/s]\n",
      "model.safetensors: 100%|██████████| 670M/670M [00:57<00:00, 11.7MB/s] \n",
      "tokenizer_config.json: 100%|██████████| 342/342 [00:00<00:00, 2.74MB/s]\n",
      "vocab.txt: 100%|██████████| 232k/232k [00:00<00:00, 584kB/s]\n",
      "tokenizer.json: 100%|██████████| 712k/712k [00:00<00:00, 10.1MB/s]\n",
      "special_tokens_map.json: 100%|██████████| 125/125 [00:00<00:00, 646kB/s]\n"
     ]
    }
   ],
   "source": [
    "# Embeddings\n",
    "\n",
    "from llama_index.embeddings.huggingface import HuggingFaceEmbedding\n",
    "\n",
    "embed_model = HuggingFaceEmbedding(model_name=\"thenlper/gte-large\", cache_folder=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the LLM, Mistral 7B Instruct (though could be any another for our purposes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes\n",
      "llama_model_loader: loaded meta data with 20 key-value pairs and 291 tensors from ./Models/mistral-7b-instruct-v0.1.Q6_K.gguf (version GGUF V2)\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = mistralai_mistral-7b-instruct-v0.1\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 32768\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 10000.000000\n",
      "llama_model_loader: - kv  11:                          general.file_type u32              = 18\n",
      "llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:   65 tensors\n",
      "llama_model_loader: - type q6_K:  226 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V2\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 32768\n",
      "llm_load_print_meta: n_embd           = 4096\n",
      "llm_load_print_meta: n_head           = 32\n",
      "llm_load_print_meta: n_head_kv        = 8\n",
      "llm_load_print_meta: n_layer          = 32\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_embd_head_k    = 128\n",
      "llm_load_print_meta: n_embd_head_v    = 128\n",
      "llm_load_print_meta: n_gqa            = 4\n",
      "llm_load_print_meta: n_embd_k_gqa     = 1024\n",
      "llm_load_print_meta: n_embd_v_gqa     = 1024\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 14336\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 32768\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 7B\n",
      "llm_load_print_meta: model ftype      = Q6_K\n",
      "llm_load_print_meta: model params     = 7.24 B\n",
      "llm_load_print_meta: model size       = 5.53 GiB (6.56 BPW) \n",
      "llm_load_print_meta: general.name     = mistralai_mistral-7b-instruct-v0.1\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size =    0.22 MiB\n",
      "llm_load_tensors: offloading 32 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 33/33 layers to GPU\n",
      "llm_load_tensors:        CPU buffer size =   102.54 MiB\n",
      "llm_load_tensors:      CUDA0 buffer size =  5563.55 MiB\n",
      "...................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 8192\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init:      CUDA0 KV buffer size =  1024.00 MiB\n",
      "llama_new_context_with_model: KV self size  = 1024.00 MiB, K (f16):  512.00 MiB, V (f16):  512.00 MiB\n",
      "llama_new_context_with_model:  CUDA_Host input buffer size   =    25.07 MiB\n",
      "llama_new_context_with_model:      CUDA0 compute buffer size =   560.03 MiB\n",
      "llama_new_context_with_model:  CUDA_Host compute buffer size =     8.00 MiB\n",
      "llama_new_context_with_model: graph splits (measure): 3\n",
      "AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | \n",
      "Model metadata: {'tokenizer.ggml.unknown_token_id': '0', 'tokenizer.ggml.eos_token_id': '2', 'general.architecture': 'llama', 'llama.rope.freq_base': '10000.000000', 'llama.context_length': '32768', 'general.name': 'mistralai_mistral-7b-instruct-v0.1', 'llama.embedding_length': '4096', 'llama.feed_forward_length': '14336', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'llama.rope.dimension_count': '128', 'tokenizer.ggml.bos_token_id': '1', 'llama.attention.head_count': '32', 'llama.block_count': '32', 'llama.attention.head_count_kv': '8', 'general.quantization_version': '2', 'tokenizer.ggml.model': 'llama', 'general.file_type': '18'}\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "from llama_index.llms.llama_cpp import LlamaCPP\n",
    "from llama_index.llms.llama_cpp.llama_utils import messages_to_prompt, completion_to_prompt\n",
    "llm = LlamaCPP(\n",
    "    model_url=None, # We'll load locally.\n",
    "    model_path='./Models/mistral-7b-instruct-v0.1.Q6_K.gguf', # 6-bit model\n",
    "    temperature=0.1,\n",
    "    max_new_tokens=1024, # Increasing to support longer responses\n",
    "    context_window=8192, # Mistral7B has an 8K context-window\n",
    "    generate_kwargs={},\n",
    "    # set to at least 1 to use GPU\n",
    "    model_kwargs={\"n_gpu_layers\": 33}, # 33 was all that was needed for this model and the RTX 3090\n",
    "    messages_to_prompt=messages_to_prompt,\n",
    "    completion_to_prompt=completion_to_prompt,\n",
    "    verbose=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create and set the tokenizer which allows us to get the tokens used in text. This needs to match the LLM used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tokenizer must match the model we're using\n",
    "\n",
    "from llama_index.core import set_global_tokenizer\n",
    "\n",
    "# huggingface\n",
    "from transformers import AutoTokenizer\n",
    "set_global_tokenizer(\n",
    "  AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-Instruct-v0.1\").encode\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a function to split text into paragraphs but keep numbered sections and bullet points together. This is suitable for the document because it has numbered and bulleted points - this would need to be changed to suit the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "# Define the regular expression pattern for splitting paragraphs\n",
    "para_split_pattern = re.compile(r'\\n\\n\\n')\n",
    "\n",
    "# Splits a document's text into paragraphs but if it has numbered or bulleted points, they will be included with the paragraph before it.\n",
    "def split_text_into_paragraphs(text):\n",
    "\n",
    "\n",
    "    # Use the pattern to split the text into paragraphs\n",
    "    paragraphs = para_split_pattern.split(text)\n",
    "\n",
    "    # Combine paragraphs that should not be split\n",
    "    combined_paragraphs = [paragraphs[0]]\n",
    "\n",
    "    for p in paragraphs[1:]:\n",
    "        # Check if the paragraph starts with a number or a dash and, if so, concatenate it to the previous paragraph so we keep them all in one chunk\n",
    "\n",
    "        # Strip out any leading new lines\n",
    "        p = p.lstrip('\\n')\n",
    "\n",
    "        if p and (p[0].isdigit() or p[0] == '-'):\n",
    "            combined_paragraphs[-1] += '\\n\\n\\n' + p\n",
    "        else:\n",
    "            combined_paragraphs.append(p)\n",
    "\n",
    "    # Remove empty strings from the result\n",
    "    combined_paragraphs = [p.strip() for p in combined_paragraphs if p.strip()]\n",
    "\n",
    "    return combined_paragraphs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create nodes from the paragraphs that we've carefully split up, counting the paragraphs so we know what kind of token length we're working with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document Thundertooth Part 1.docx has 12 paragraphs, token lengths:\n",
      "[5, 102, 78, 98, 70, 88, 72, 29, 73, 56, 67, 118]\n",
      "Document Thundertooth Part 2.docx has 10 paragraphs, token lengths:\n",
      "[5, 97, 79, 71, 93, 61, 75, 73, 74, 76]\n",
      "Document Thundertooth Part 3.docx has 10 paragraphs, token lengths:\n",
      "[5, 92, 58, 62, 228, 80, 106, 82, 73, 97]\n",
      "\n",
      "** The maximum paragraph tokens is 228 **\n",
      "\n",
      "** The average paragraph's token count is 76 **\n",
      "\n",
      "** Created 32 nodes **\n"
     ]
    }
   ],
   "source": [
    "from llama_index.core.utilities.token_counting import TokenCounter\n",
    "from llama_index.core.schema import TextNode\n",
    "import uuid\n",
    "\n",
    "token_counter = TokenCounter() # Uses the global tokenizer set above, which should match the LLM\n",
    "\n",
    "paragraph_separator = \"\\n\\n\\n\"\n",
    "\n",
    "# Stores the maximum length of a paragraph, in tokens\n",
    "max_paragraph_tokens = 0\n",
    "\n",
    "# Total tokens, used to determine average\n",
    "total_paragraph_tokens = 0\n",
    "\n",
    "# Nodes\n",
    "paragraph_nodes = []\n",
    "\n",
    "# Loop through the documents, splitting each into paragraphs and checking the number of tokens per paragraph\n",
    "for document in documents:\n",
    "\n",
    "    paragraph_token_lens = []\n",
    "    # paragraphs = document.text.split(paragraph_separator)\n",
    "    paragraphs = split_text_into_paragraphs(document.text)\n",
    "    print(f\"Document {document.metadata['file_name']} has {len(paragraphs)} paragraphs, token lengths:\")\n",
    "    for paragraph in paragraphs:\n",
    "        token_count = token_counter.get_string_tokens(paragraph)\n",
    "        paragraph_token_lens.append(token_count)\n",
    "        # print(f\"Paragraph tokens: {token_count}\")\n",
    "\n",
    "        if token_count > max_paragraph_tokens:\n",
    "            max_paragraph_tokens = token_count\n",
    "\n",
    "        total_paragraph_tokens = total_paragraph_tokens + token_count\n",
    "\n",
    "        # Create and add the node from the paragraph\n",
    "        # include metadata we can use for citations\n",
    "        node = TextNode(text=paragraph, id=uuid.uuid4())\n",
    "        node.metadata[\"document_name\"] =document.metadata[\"file_name\"]\n",
    "        node.metadata[\"token_count\"] = token_count\n",
    "        paragraph_nodes.append(node)\n",
    "\n",
    "    print(paragraph_token_lens)\n",
    "\n",
    "print(f\"\\n** The maximum paragraph tokens is {max_paragraph_tokens} **\")\n",
    "\n",
    "average_paragraph_tokens = int(total_paragraph_tokens / len(paragraph_nodes))\n",
    "print(f\"\\n** The average paragraph's token count is {average_paragraph_tokens} **\")\n",
    "\n",
    "print(f\"\\n** Created {len(paragraph_nodes)} nodes **\")\n",
    "# paragraph_nodes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the maximum paragraph token count is 228 and the average is 76. This helps us work out how many paragraphs we can return for the LLM to use for RAG.\n",
    "\n",
    "If we have 1000 tokens to feed into the LLM we can return 10 or so pargraphs, that gives a good amount of context for the LLM to work with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import Settings\n",
    "\n",
    "Settings.llm = llm\n",
    "Settings.embed_model = embed_model\n",
    "# chunk_size=500, # The token chunk size for each chunk. ** we are creating chunks automatically so no need to set this **\n",
    "# chunk_overlap=5, # The token overlap of each chunk when splitting. ** we are creating chunks automatically so no need to set this **\n",
    "Settings.num_output = 1000 # Let's allow up to 1000 tokens to be output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "service_context.node_parser:\n",
      "Chunk Overlap: 200\n",
      "Chunk Size: 1024\n",
      "Paragraph Separator: \\n\\n\\n\n",
      "Secondary Chunking RegEx: [^,.;。？！]+[,.;。？！]?\n",
      "Include Metadata: True\n",
      "\n",
      "service_context.prompt_helper:\n",
      "Context Window: 8192\n",
      "Chunk Overlap Ratio: 0.1\n",
      "Chunk Size Limit: None\n",
      "Chunk Separator: '   '\n"
     ]
    }
   ],
   "source": [
    "# Let's look at the chunking and prompting parameters from our service_context\n",
    "\n",
    "print(\"service_context.node_parser:\")\n",
    "print(\"Chunk Overlap:\",Settings.node_parser.chunk_overlap)\n",
    "print(\"Chunk Size:\",Settings.node_parser.chunk_size)\n",
    "print(\"Paragraph Separator:\",Settings.node_parser.paragraph_separator.replace(\"\\n\",\"\\\\n\"))\n",
    "print(\"Secondary Chunking RegEx:\",Settings.node_parser.secondary_chunking_regex)\n",
    "print(\"Include Metadata:\",Settings.node_parser.include_metadata)\n",
    "\n",
    "print(\"\\nservice_context.prompt_helper:\")\n",
    "print(\"Context Window:\", Settings.prompt_helper.context_window) # The maximum context size that will get sent to the LLM.\n",
    "print(\"Chunk Overlap Ratio:\", Settings.prompt_helper.chunk_overlap_ratio) # The percentage token amount that each chunk should overlap.\n",
    "print(\"Chunk Size Limit:\", Settings.prompt_helper.chunk_size_limit) # The maximum size of a chunk.\n",
    "print(\"Chunk Separator: '\", Settings.prompt_helper.separator, \"'\") # The separator when chunking tokens."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create the vector store\n",
    "\n",
    "Importantly this is created from the nodes we have created, rather than being chunked automatically"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating embeddings: 100%|██████████| 32/32 [00:00<00:00, 35.96it/s]\n"
     ]
    }
   ],
   "source": [
    "# Now we need to index these nodes, putting them into the vector store\n",
    "\n",
    "from llama_index.core import VectorStoreIndex\n",
    "\n",
    "index_from_nodes = VectorStoreIndex(paragraph_nodes, show_progress=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- 0 ---\n",
      "Thundertooth Part 1.docx\n",
      "id: da71c044-785c-44e8-8f0f-a8449768d62d\n",
      "characters: 12\n",
      "tokens: 5\n",
      "[Text Start]\n",
      "Thundertooth\n",
      "[Text End]\n",
      "\n",
      "--- 1 ---\n",
      "Thundertooth Part 1.docx\n",
      "id: b9ef759e-4943-45eb-a2a2-f3f31fb8ab12\n",
      "characters: 428\n",
      "tokens: 102\n",
      "[Text Start]\n",
      "Once upon a time, in a prehistoric land filled with dense forests and roaring rivers, there lived a dinosaur named Thundertooth. Thundertooth was no ordinary dinosaur; he possessed the rare ability to speak, a talent that set him apart from his ancient companions. One fateful day, as Thundertooth was basking in the warmth of the sun, a mysterious portal opened before him, and he found himself hurtling through time and space.\n",
      "[Text End]\n",
      "\n",
      "--- 2 ---\n",
      "Thundertooth Part 1.docx\n",
      "id: 00956141-f2ed-415c-972f-d31f8b3ec76f\n",
      "characters: 330\n",
      "tokens: 78\n",
      "[Text Start]\n",
      "As the dazzling vortex subsided, Thundertooth opened his eyes to a world unlike anything he had ever seen. The air was filled with the hum of engines, and towering structures reached towards the sky. Thundertooth's surroundings were a blend of metal and glass, and he quickly realized that he had been transported to a future era.\n",
      "[Text End]\n",
      "\n",
      "--- 3 ---\n",
      "Thundertooth Part 1.docx\n",
      "id: dcf2beaa-56a8-4d0e-a001-23c74dcc7ac3\n",
      "characters: 395\n",
      "tokens: 98\n",
      "[Text Start]\n",
      "The once mighty dinosaur now stood bewildered in the midst of a bustling city. Above him, sleek flying cars zipped through the air, leaving trails of neon lights in their wake. Thundertooth felt like an ancient relic in this technological jungle, lost and out of place. With each step, he marveled at the skyscrapers that loomed overhead, their surfaces reflecting the myriad lights of the city.\n",
      "[Text End]\n",
      "\n",
      "--- 4 ---\n",
      "Thundertooth Part 1.docx\n",
      "id: f830d9c8-783f-421d-b874-8da208534c74\n",
      "characters: 294\n",
      "tokens: 70\n",
      "[Text Start]\n",
      "However, as night fell, Thundertooth's stomach growled loudly. He realized that he was hungry, and the once vibrant city now seemed like a daunting maze of unfamiliar smells and sights. He wandered through the streets, his massive form drawing astonished stares from the futuristic inhabitants.\n",
      "[Text End]\n",
      "\n",
      "--- 5 ---\n",
      "Thundertooth Part 1.docx\n",
      "id: 1f1c22b8-22d2-42b5-a451-f5005ba66c86\n",
      "characters: 333\n",
      "tokens: 88\n",
      "[Text Start]\n",
      "Thundertooth faced a dilemma – he was hungry, but he couldn't bring himself to feast on the humans who scurried around like ants. As his hunger grew, he stumbled upon a park, an oasis of greenery amidst the concrete and steel. The park was adorned with holographic flowers that emitted a gentle glow, creating an ethereal atmosphere.\n",
      "[Text End]\n",
      "\n",
      "--- 6 ---\n",
      "Thundertooth Part 1.docx\n",
      "id: 696f99de-17d1-4a43-b5eb-647cfb8f9dd2\n",
      "characters: 296\n",
      "tokens: 72\n",
      "[Text Start]\n",
      "While Thundertooth marveled at the beauty of the park, the mayor of the city happened to be passing by. Mayor Eleanor Grace, a charismatic and forward-thinking leader, was immediately intrigued by the sight of the talking dinosaur. She approached Thundertooth with a mix of curiosity and caution.\n",
      "[Text End]\n",
      "\n",
      "--- 7 ---\n",
      "Thundertooth Part 1.docx\n",
      "id: 6e415b4c-154d-4dd7-bcdf-6b0f3fd5b8a8\n",
      "characters: 115\n",
      "tokens: 29\n",
      "[Text Start]\n",
      "\"Hello there, majestic creature. What brings you to our time?\" Mayor Grace inquired, her voice calm and reassuring.\n",
      "[Text End]\n",
      "\n",
      "--- 8 ---\n",
      "Thundertooth Part 1.docx\n",
      "id: 641fb9ae-8d80-4ce6-b5ae-a051a47b7895\n",
      "characters: 315\n",
      "tokens: 73\n",
      "[Text Start]\n",
      "Thundertooth, though initially startled, found comfort in the mayor's soothing tone. In broken sentences, he explained his journey through time, the strange portal, and his hunger dilemma. Mayor Grace listened intently, her eyes widening with amazement at the tale of the prehistoric dinosaur navigating the future.\n",
      "[Text End]\n",
      "\n",
      "--- 9 ---\n",
      "Thundertooth Part 1.docx\n",
      "id: 92e51777-94e9-4d6a-8773-8819d497cda9\n",
      "characters: 228\n",
      "tokens: 56\n",
      "[Text Start]\n",
      "Realizing the dinosaur's predicament, Mayor Grace extended an invitation. \"You are welcome in our city, Thundertooth. We can find a way to provide for you without causing harm to anyone. Let us work together to find a solution.\"\n",
      "[Text End]\n",
      "\n",
      "--- 10 ---\n",
      "Thundertooth Part 1.docx\n",
      "id: f0088cb3-b22d-4ec1-b71b-894ebf2f4958\n",
      "characters: 305\n",
      "tokens: 67\n",
      "[Text Start]\n",
      "Grateful for the mayor's hospitality, Thundertooth followed her through the city. Together, they explored the futuristic marketplaces and innovative food labs, eventually discovering a sustainable solution that satisfied the dinosaur's hunger without compromising the well-being of the city's inhabitants.\n",
      "[Text End]\n",
      "\n",
      "--- 11 ---\n",
      "Thundertooth Part 1.docx\n",
      "id: 487186a4-2dd5-42d6-94dd-de7a25955745\n",
      "characters: 534\n",
      "tokens: 118\n",
      "[Text Start]\n",
      "As the news of Thundertooth's arrival spread, the city embraced the talking dinosaur as a symbol of unity between the past and the future. Thundertooth found a new home in the city's park, where holographic flowers bloomed, and the citizens marveled at the beauty of coexistence across time. And so, in this extraordinary city of flying cars and advanced technology, Thundertooth became a beloved figure, a living bridge between eras, teaching the people that understanding and cooperation could overcome even the greatest challenges.\n",
      "[Text End]\n",
      "\n",
      "--- 12 ---\n",
      "Thundertooth Part 2.docx\n",
      "id: 1a0aa5f9-0c6c-4abf-a32b-271828361b00\n",
      "characters: 12\n",
      "tokens: 5\n",
      "[Text Start]\n",
      "Thundertooth\n",
      "[Text End]\n",
      "\n",
      "--- 13 ---\n",
      "Thundertooth Part 2.docx\n",
      "id: 34dd461a-13da-4621-ba34-a08e8b63067a\n",
      "characters: 465\n",
      "tokens: 97\n",
      "[Text Start]\n",
      "Embraced by the futuristic city and its inhabitants, Thundertooth found a sense of purpose beyond merely satisfying his hunger. Inspired by the advanced technology surrounding him, he decided to channel his creativity into something extraordinary. With the help of the city's brilliant engineers, Thundertooth founded a one-of-a-kind toy factory that produced amazing widgets – magical, interactive toys that captivated the hearts of both children and adults alike.\n",
      "[Text End]\n",
      "\n",
      "--- 14 ---\n",
      "Thundertooth Part 2.docx\n",
      "id: 963f1111-426c-4d3b-9cd5-ebcc9acf9fa8\n",
      "characters: 373\n",
      "tokens: 79\n",
      "[Text Start]\n",
      "Thundertooth's toy factory became a sensation, and its creations were highly sought after. The widgets incorporated cutting-edge holographic displays, levitation technology, and even the ability to change shapes and colors with a mere thought. Children across the city rejoiced as they played with these incredible toys that seemed to bring their wildest fantasies to life.\n",
      "[Text End]\n",
      "\n",
      "--- 15 ---\n",
      "Thundertooth Part 2.docx\n",
      "id: d12918c3-8bc5-43c0-b67b-5be14222735a\n",
      "characters: 302\n",
      "tokens: 71\n",
      "[Text Start]\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "[Text End]\n",
      "\n",
      "--- 16 ---\n",
      "Thundertooth Part 2.docx\n",
      "id: 5d099199-0d3d-4877-a2e2-e61c9b50a059\n",
      "characters: 384\n",
      "tokens: 93\n",
      "[Text Start]\n",
      "Lumina: The eldest of Thundertooth's children, Lumina inherited her mother's intelligence and her father's sense of wonder. With sparkling scales that emitted a soft glow, Lumina had the ability to generate light at will. She became fascinated with technology, often spending hours tinkering with gadgets and inventing new ways to enhance the widgets produced in the family's factory.\n",
      "[Text End]\n",
      "\n",
      "--- 17 ---\n",
      "Thundertooth Part 2.docx\n",
      "id: d5102d0c-4b22-4d5e-89c7-6eae9721e463\n",
      "characters: 271\n",
      "tokens: 61\n",
      "[Text Start]\n",
      "Echo: The second-born, Echo, had a gift for mimicry. He could perfectly replicate any sound or voice he heard, providing entertainment to the entire city. His playful nature and ability to bring joy to those around him made him a favorite among the neighborhood children.\n",
      "[Text End]\n",
      "\n",
      "--- 18 ---\n",
      "Thundertooth Part 2.docx\n",
      "id: f4a5c21b-3641-4fcf-a2d1-d2ebf3b8d81a\n",
      "characters: 312\n",
      "tokens: 75\n",
      "[Text Start]\n",
      "Sapphire: Sapphire, the third sibling, had scales that shimmered like precious gems. She possessed a unique talent for calming and healing, a trait she inherited from both her parents. Whenever someone in the city felt stressed or unwell, Sapphire would extend her gentle touch, bringing comfort and tranquility.\n",
      "[Text End]\n",
      "\n",
      "--- 19 ---\n",
      "Thundertooth Part 2.docx\n",
      "id: 3cd6c2db-f4cc-4519-b42b-12c245788588\n",
      "characters: 331\n",
      "tokens: 73\n",
      "[Text Start]\n",
      "Ignis: The youngest of the family, Ignis, had fiery red scales that hinted at his exceptional ability – the power to control small flames. While initially seen as a potential hazard, Ignis channeled his fiery talents into creating mesmerizing light shows, becoming the city's favorite entertainer during festivals and celebrations.\n",
      "[Text End]\n",
      "\n",
      "--- 20 ---\n",
      "Thundertooth Part 2.docx\n",
      "id: d2855ae7-c17a-43de-b559-775da4eff423\n",
      "characters: 335\n",
      "tokens: 74\n",
      "[Text Start]\n",
      "Thundertooth and Seraphina reveled in the joy of parenthood, watching their children grow and flourish in the futuristic landscape they now called home. The family became an integral part of the city's fabric, not only through the widgets produced in their factory but also through the positive impact each member had on the community.\n",
      "[Text End]\n",
      "\n",
      "--- 21 ---\n",
      "Thundertooth Part 2.docx\n",
      "id: 8c7eda7c-cdb6-45d7-995f-109155ca6186\n",
      "characters: 375\n",
      "tokens: 76\n",
      "[Text Start]\n",
      "The toy factory became a symbol of innovation and unity, bringing together dinosaurs and humans in a shared appreciation for creativity and wonder. Thundertooth's legacy extended beyond his time-traveling adventure, leaving an indelible mark on the city and its inhabitants, reminding them that sometimes, the most magical things could emerge from the most unexpected places.\n",
      "[Text End]\n",
      "\n",
      "--- 22 ---\n",
      "Thundertooth Part 3.docx\n",
      "id: fe3d6733-828d-4cd8-ab2c-c7c4aab5c249\n",
      "characters: 12\n",
      "tokens: 5\n",
      "[Text Start]\n",
      "Thundertooth\n",
      "[Text End]\n",
      "\n",
      "--- 23 ---\n",
      "Thundertooth Part 3.docx\n",
      "id: 65aa164f-6870-4c2f-889f-51601adb0cd5\n",
      "characters: 416\n",
      "tokens: 92\n",
      "[Text Start]\n",
      "One fateful day, as the citizens of the futuristic city went about their daily lives, a collective gasp echoed through the streets as a massive meteor hurtled towards Earth. Panic spread like wildfire as people looked to the sky in horror, realizing the impending catastrophe. The city's advanced technology detected the threat, and an emergency broadcast echoed through the streets, urging everyone to seek shelter.\n",
      "[Text End]\n",
      "\n",
      "--- 24 ---\n",
      "Thundertooth Part 3.docx\n",
      "id: 86eb1cd1-b9c9-479e-ab68-5ecaea981b31\n",
      "characters: 259\n",
      "tokens: 58\n",
      "[Text Start]\n",
      "Thundertooth, ever the protector of his newfound home, wasted no time. With a determined gleam in his eyes, he gathered his family and hurried to the city's command center, where Mayor Grace and the leading scientists were coordinating the evacuation efforts.\n",
      "[Text End]\n",
      "\n",
      "--- 25 ---\n",
      "Thundertooth Part 3.docx\n",
      "id: fa379c08-ec83-47b8-ad7b-21ce3ae9cfce\n",
      "characters: 254\n",
      "tokens: 62\n",
      "[Text Start]\n",
      "The mayor, recognizing Thundertooth's intelligence and resourcefulness, approached him. \"Thundertooth, we need a plan to divert or neutralize the meteor. Our technology can only do so much, but with your unique abilities, perhaps we can find a solution.\"\n",
      "[Text End]\n",
      "\n",
      "--- 26 ---\n",
      "Thundertooth Part 3.docx\n",
      "id: 4127dce1-4a8b-42b4-aee1-9dafa160aa4c\n",
      "characters: 923\n",
      "tokens: 228\n",
      "[Text Start]\n",
      "Thundertooth nodded, understanding the gravity of the situation. He gathered Lumina, Echo, Sapphire, and Ignis, explaining the urgency and the role each of them would play in the impending crisis.\n",
      "\n",
      "\n",
      "1. **Lumina**: Utilizing her deep understanding of technology, Lumina would enhance the city's energy systems to generate a powerful force field, providing a protective barrier against the meteor's impact.\n",
      "\n",
      "\n",
      "2. **Echo**: With his extraordinary mimicry abilities, Echo would amplify the emergency signals, ensuring that every citizen received timely warnings and instructions for evacuation.\n",
      "\n",
      "\n",
      "3. **Sapphire**: Harnessing her calming and healing powers, Sapphire would assist in calming the panicked masses, ensuring an orderly and efficient evacuation.\n",
      "\n",
      "\n",
      "4. **Ignis**: Drawing upon his fiery talents, Ignis would create controlled bursts of heat, attempting to alter the meteor's trajectory and reduce its destructive force.\n",
      "[Text End]\n",
      "\n",
      "--- 27 ---\n",
      "Thundertooth Part 3.docx\n",
      "id: 19b5a719-25ad-455a-9e2d-f7ff83cf8778\n",
      "characters: 345\n",
      "tokens: 80\n",
      "[Text Start]\n",
      "As the citizens evacuated to designated shelters, the Thundertooth family sprang into action. Lumina worked tirelessly to strengthen the city's energy systems, Echo echoed evacuation orders through the city's speakers, Sapphire offered comfort to those in distress, and Ignis unleashed controlled bursts of flames towards the approaching meteor.\n",
      "[Text End]\n",
      "\n",
      "--- 28 ---\n",
      "Thundertooth Part 3.docx\n",
      "id: aa15dfd0-020a-4da6-9d4e-e9b46ef5ec59\n",
      "characters: 417\n",
      "tokens: 106\n",
      "[Text Start]\n",
      "Thundertooth stood at the forefront, using his mighty roar to coordinate and inspire the efforts of the city's inhabitants. The ground trembled as the meteor drew closer, but the Thundertooth family's coordinated efforts began to take effect. Lumina's force field shimmered to life, deflecting the meteor's deadly path. Echo's amplified warnings reached every corner of the city, ensuring that no one was left behind.\n",
      "[Text End]\n",
      "\n",
      "--- 29 ---\n",
      "Thundertooth Part 3.docx\n",
      "id: 8fb5752d-b710-400c-8183-0f6ee658cae2\n",
      "characters: 340\n",
      "tokens: 82\n",
      "[Text Start]\n",
      "As Ignis's controlled bursts of flames interacted with the meteor, it began to change course. The combined efforts of the Thundertooth family, guided by their unique talents, diverted the catastrophic collision. The meteor, once destined for destruction, now harmlessly sailed past the Earth, leaving the city and its inhabitants unscathed.\n",
      "[Text End]\n",
      "\n",
      "--- 30 ---\n",
      "Thundertooth Part 3.docx\n",
      "id: f132e795-5f1a-4d12-bc4e-90f11c91ccad\n",
      "characters: 306\n",
      "tokens: 73\n",
      "[Text Start]\n",
      "The citizens, emerging from their shelters, erupted into cheers of gratitude. Mayor Grace approached Thundertooth, expressing her heartfelt thanks for the family's heroic efforts. The Thundertooth family, tired but triumphant, basked in the relief of having saved their beloved city from imminent disaster.\n",
      "[Text End]\n",
      "\n",
      "--- 31 ---\n",
      "Thundertooth Part 3.docx\n",
      "id: 7de06485-91a0-435b-9956-c7df5ce4599b\n",
      "characters: 440\n",
      "tokens: 97\n",
      "[Text Start]\n",
      "In the wake of the crisis, the citizens of the futuristic city hailed Thundertooth and his family as true heroes. The toy factory that once brought joy to children now became a symbol of resilience and unity. The Thundertooth family's legacy was forever etched in the city's history, a testament to the power of cooperation and the extraordinary capabilities that could emerge when dinosaurs and humans worked together for the greater good.\n",
      "[Text End]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# List \"documents\" stored in our vector store - we created these and stored them\n",
    "\n",
    "for i, document_id in enumerate(index_from_nodes.docstore.docs):\n",
    "    document = index_from_nodes.docstore.get_document(document_id)\n",
    "    print(f\"--- {i} ---\\n{document.extra_info['document_name']}\")\n",
    "    print(f\"id: {document.node_id}\")\n",
    "    print(f\"characters: {len(document.text)}\")\n",
    "    print(f\"tokens: {document.extra_info['token_count']}\")\n",
    "    print(f\"[Text Start]\\n{document.text}\\n[Text End]\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many paragraphs should we return? Let's assume we have 1,000 tokens to work with and we know the average paragraph token length is 76."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We'll return 10 paragraphs for context\n"
     ]
    }
   ],
   "source": [
    "import math\n",
    "\n",
    "working_context_token_length = 1000     # Let's allow 1,000 tokens for context\n",
    "\n",
    "# Round up to the closest 50 tokens for the average, resulting in 100 tokens\n",
    "return_paragraphs = int(working_context_token_length / (math.ceil(average_paragraph_tokens / 50) * 50))\n",
    "\n",
    "print(f\"We'll return {return_paragraphs} paragraphs for context\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test time. We run a question and check if it returned the number of paragraphs we allowed it to and whether it could answer the question correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =     139.50 ms\n",
      "llama_print_timings:      sample time =      10.36 ms /    39 runs   (    0.27 ms per token,  3765.21 tokens per second)\n",
      "llama_print_timings: prompt eval time =     428.18 ms /  1419 tokens (    0.30 ms per token,  3314.00 tokens per second)\n",
      "llama_print_timings:        eval time =     433.16 ms /    38 runs   (   11.40 ms per token,    87.73 tokens per second)\n",
      "llama_print_timings:       total time =     941.98 ms /  1457 tokens\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/10: |Thundertooth Part 2.docx| Source 1:\n",
      "Thundertooth and Seraphina reveled in the joy of parenthood, watching their children grow and flourish in the futuristic landscape they now called home. The family became an integral part of the city's fabric, not only through the widgets produced in their factory but also through the positive impact each member had on the community.\n",
      "\n",
      "2/10: |Thundertooth Part 2.docx| Source 2:\n",
      "Lumina: The eldest of Thundertooth's children, Lumina inherited her mother's intelligence and her father's sense of wonder. With sparkling scales that emitted a soft glow, Lumina had the ability to generate light at will. She became fascinated with technology, often spending hours tinkering with gadgets and inventing new ways to enhance the widgets produced in the family's factory.\n",
      "\n",
      "3/10: |Thundertooth Part 2.docx| Source 3:\n",
      "Embraced by the futuristic city and its inhabitants, Thundertooth found a sense of purpose beyond merely satisfying his hunger. Inspired by the advanced technology surrounding him, he decided to channel his creativity into something extraordinary. With the help of the city's brilliant engineers, Thundertooth founded a one-of-a-kind toy factory that produced amazing widgets – magical, interactive toys that captivated the hearts of both children and adults alike.\n",
      "\n",
      "4/10: |Thundertooth Part 3.docx| Source 4:\n",
      "In the wake of the crisis, the citizens of the futuristic city hailed Thundertooth and his family as true heroes. The toy factory that once brought joy to children now became a symbol of resilience and unity. The Thundertooth family's legacy was forever etched in the city's history, a testament to the power of cooperation and the extraordinary capabilities that could emerge when dinosaurs and humans worked together for the greater good.\n",
      "\n",
      "5/10: |Thundertooth Part 2.docx| Source 5:\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "6/10: |Thundertooth Part 2.docx| Source 6:\n",
      "Thundertooth's toy factory became a sensation, and its creations were highly sought after. The widgets incorporated cutting-edge holographic displays, levitation technology, and even the ability to change shapes and colors with a mere thought. Children across the city rejoiced as they played with these incredible toys that seemed to bring their wildest fantasies to life.\n",
      "\n",
      "7/10: |Thundertooth Part 3.docx| Source 7:\n",
      "The citizens, emerging from their shelters, erupted into cheers of gratitude. Mayor Grace approached Thundertooth, expressing her heartfelt thanks for the family's heroic efforts. The Thundertooth family, tired but triumphant, basked in the relief of having saved their beloved city from imminent disaster.\n",
      "\n",
      "8/10: |Thundertooth Part 3.docx| Source 8:\n",
      "As the citizens evacuated to designated shelters, the Thundertooth family sprang into action. Lumina worked tirelessly to strengthen the city's energy systems, Echo echoed evacuation orders through the city's speakers, Sapphire offered comfort to those in distress, and Ignis unleashed controlled bursts of flames towards the approaching meteor.\n",
      "\n",
      "9/10: |Thundertooth Part 1.docx| Source 9:\n",
      "As the news of Thundertooth's arrival spread, the city embraced the talking dinosaur as a symbol of unity between the past and the future. Thundertooth found a new home in the city's park, where holographic flowers bloomed, and the citizens marveled at the beauty of coexistence across time. And so, in this extraordinary city of flying cars and advanced technology, Thundertooth became a beloved figure, a living bridge between eras, teaching the people that understanding and cooperation could overcome even the greatest challenges.\n",
      "\n",
      "10/10: |Thundertooth Part 2.docx| Source 10:\n",
      "Echo: The second-born, Echo, had a gift for mimicry. He could perfectly replicate any sound or voice he heard, providing entertainment to the entire city. His playful nature and ability to bring joy to those around him made him a favorite among the neighborhood children.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from llama_index.core.query_engine import CitationQueryEngine\n",
    "query_engine = CitationQueryEngine.from_args(\n",
    "    index_from_nodes,\n",
    "    similarity_top_k=return_paragraphs,\n",
    ")\n",
    "\n",
    "# For citations we get the document info\n",
    "DB_DOC_ID_KEY = \"db_document_id\"\n",
    "\n",
    "test_question = \"Did they have any children? If so, what were their names?\"\n",
    "\n",
    "queryQuestion = \"<s>[INST] You are a technology specialist. Answer questions in a positive, helpful and empathetic way. Answer the following question: \" + test_question + \" [/INST]\"\n",
    "\n",
    "response = query_engine.query(queryQuestion)\n",
    "\n",
    "for index, node in enumerate(response.source_nodes, start=1):\n",
    "    print(f\"{index}/{len(response.source_nodes)}: |{node.node.metadata['document_name']}| {node.node.get_text()}\")\n",
    "\n",
    "test_response = str(response.response).strip()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here's the question and result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Question:\n",
      "Did they have any children? If so, what were their names?\n",
      "\n",
      "\n",
      "Response:\n",
      "Based on the provided sources, Thundertooth and Seraphina had four children together. Their names were Lumina, Echo, Sapphire, and Ignis.\n"
     ]
    }
   ],
   "source": [
    "print(f\"\\n\\nQuestion:\\n{test_question}\")\n",
    "print(f\"\\n\\nResponse:\\n{test_response}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks good!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "LlamaIndexGeneric",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}