LlamaIndex_Phi-2-FP16-RAG.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Phi-2 (FP16)\n",
    "#### NOTE: YOU WILL NEED LLAMA-CPP-PYTHON VERSION v0.2.24 OR HIGHER\n",
    "\n",
    "This notebook demonstrates the use of LlamaIndex for Retrieval Augmented Generation in Linux and with Nvidia's CUDA.\n",
    "\n",
    "See the [README.md](README.md) file for help on how to run this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Prepare Llama Index for use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/mark/miniconda3/envs/LlamaIndexRAGLinux/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "import logging\n",
    "import sys\n",
    "\n",
    "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
    "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))\n",
    "\n",
    "from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, ServiceContext"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Load the Word document(s)\n",
    "\n",
    "Note: A fictitious story about Thundertooth a dinosaur who has travelled to the future. Thanks ChatGPT!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents = SimpleDirectoryReader(\"./Data/\").load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Instantiate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes\n",
      "llama_model_loader: loaded meta data with 18 key-value pairs and 325 tensors from ./Models/phi-2-ggml-model-f16.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = phi2\n",
      "llama_model_loader: - kv   1:                               general.name str              = Phi2\n",
      "llama_model_loader: - kv   2:                        phi2.context_length u32              = 2048\n",
      "llama_model_loader: - kv   3:                      phi2.embedding_length u32              = 2560\n",
      "llama_model_loader: - kv   4:                   phi2.feed_forward_length u32              = 10240\n",
      "llama_model_loader: - kv   5:                           phi2.block_count u32              = 32\n",
      "llama_model_loader: - kv   6:                  phi2.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   7:               phi2.attention.head_count_kv u32              = 32\n",
      "llama_model_loader: - kv   8:          phi2.attention.layer_norm_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv   9:                  phi2.rope.dimension_count u32              = 32\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = gpt2\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,51200]   = [\"!\", \"\\\"\", \"#\", \"$\", \"%\", \"&\", \"'\", ...\n",
      "llama_model_loader: - kv  13:                  tokenizer.ggml.token_type arr[i32,51200]   = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...\n",
      "llama_model_loader: - kv  14:                      tokenizer.ggml.merges arr[str,50000]   = [\"Ġ t\", \"Ġ a\", \"h e\", \"i n\", \"r e\",...\n",
      "llama_model_loader: - kv  15:                tokenizer.ggml.bos_token_id u32              = 50256\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.eos_token_id u32              = 50256\n",
      "llama_model_loader: - kv  17:            tokenizer.ggml.unknown_token_id u32              = 50256\n",
      "llama_model_loader: - type  f32:  195 tensors\n",
      "llama_model_loader: - type  f16:  130 tensors\n",
      "llm_load_vocab: mismatch in special tokens definition ( 910/51200 vs 944/51200 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = phi2\n",
      "llm_load_print_meta: vocab type       = BPE\n",
      "llm_load_print_meta: n_vocab          = 51200\n",
      "llm_load_print_meta: n_merges         = 50000\n",
      "llm_load_print_meta: n_ctx_train      = 2048\n",
      "llm_load_print_meta: n_embd           = 2560\n",
      "llm_load_print_meta: n_head           = 32\n",
      "llm_load_print_meta: n_head_kv        = 32\n",
      "llm_load_print_meta: n_layer          = 32\n",
      "llm_load_print_meta: n_rot            = 32\n",
      "llm_load_print_meta: n_embd_head_k    = 80\n",
      "llm_load_print_meta: n_embd_head_v    = 80\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: n_embd_k_gqa     = 2560\n",
      "llm_load_print_meta: n_embd_v_gqa     = 2560\n",
      "llm_load_print_meta: f_norm_eps       = 1.0e-05\n",
      "llm_load_print_meta: f_norm_rms_eps   = 0.0e+00\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 10240\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 2048\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 3B\n",
      "llm_load_print_meta: model ftype      = F16\n",
      "llm_load_print_meta: model params     = 2.78 B\n",
      "llm_load_print_meta: model size       = 5.18 GiB (16.01 BPW) \n",
      "llm_load_print_meta: general.name     = Phi2\n",
      "llm_load_print_meta: BOS token        = 50256 '<|endoftext|>'\n",
      "llm_load_print_meta: EOS token        = 50256 '<|endoftext|>'\n",
      "llm_load_print_meta: UNK token        = 50256 '<|endoftext|>'\n",
      "llm_load_print_meta: LF token         = 128 'Ä'\n",
      "llm_load_tensors: ggml ctx size =    0.25 MiB\n",
      "llm_load_tensors: offloading 32 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 33/33 layers to GPU\n",
      "llm_load_tensors:        CPU buffer size =   250.00 MiB\n",
      "llm_load_tensors:      CUDA0 buffer size =  5053.65 MiB\n",
      ".............................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 2048\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init:      CUDA0 KV buffer size =   640.00 MiB\n",
      "llama_new_context_with_model: KV self size  =  640.00 MiB, K (f16):  320.00 MiB, V (f16):  320.00 MiB\n",
      "llama_new_context_with_model:  CUDA_Host input buffer size   =    10.02 MiB\n",
      "llama_new_context_with_model:      CUDA0 compute buffer size =   167.01 MiB\n",
      "llama_new_context_with_model:  CUDA_Host compute buffer size =     5.00 MiB\n",
      "llama_new_context_with_model: graph splits (measure): 3\n",
      "AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | \n",
      "Model metadata: {'tokenizer.ggml.unknown_token_id': '50256', 'tokenizer.ggml.eos_token_id': '50256', 'general.architecture': 'phi2', 'general.name': 'Phi2', 'phi2.context_length': '2048', 'tokenizer.ggml.model': 'gpt2', 'phi2.embedding_length': '2560', 'phi2.attention.head_count': '32', 'phi2.attention.head_count_kv': '32', 'phi2.feed_forward_length': '10240', 'phi2.attention.layer_norm_epsilon': '0.000010', 'phi2.block_count': '32', 'phi2.rope.dimension_count': '32', 'tokenizer.ggml.bos_token_id': '50256', 'general.file_type': '1'}\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "from llama_index.llms.llama_cpp import LlamaCPP\n",
    "from llama_index.llms.llama_cpp.llama_utils import messages_to_prompt, completion_to_prompt\n",
    "llm = LlamaCPP(\n",
    "    model_url=None, # We'll load locally.\n",
    "    model_path='./Models/phi-2-ggml-model-f16.gguf', # FP16 version of Phi-2\n",
    "    temperature=0.1,\n",
    "    max_new_tokens=1024,\n",
    "    context_window=2048, # Phi-2 2K context window - this could be a limitation for RAG as it has to put the content into this context window\n",
    "    generate_kwargs={},\n",
    "    # set to at least 1 to use GPU\n",
    "    model_kwargs={\"n_gpu_layers\": 33}, # 33 layers were all that were required and this fit within the RTX 3090's 24GB\n",
    "    messages_to_prompt=messages_to_prompt,\n",
    "    completion_to_prompt=completion_to_prompt,\n",
    "    verbose=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Checkpoint\n",
    "\n",
    "Are you running on GPU? The above output should include near the top something like:\n",
    "> ggml_init_cublas: found 1 CUDA devices:\n",
    "\n",
    "And in the full text near the bottom should be:\n",
    "> llm_load_tensors: using CUDA for GPU acceleration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Embeddings\n",
    "\n",
    "Convert your source document text into embeddings.\n",
    "\n",
    "The embedding model is from huggingface, this one performs well.\n",
    "\n",
    "> https://huggingface.co/thenlper/gte-large\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.embeddings.huggingface import HuggingFaceEmbedding\n",
    "\n",
    "embed_model = HuggingFaceEmbedding(model_name=\"thenlper/gte-large\", cache_folder=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. Prompt Template\n",
    "\n",
    "Prompt template for Phi-2 is below. As there's only a prompt we will combine the system message and prompt into the prompt.\n",
    "\n",
    "Instruct: {prompt}<br>\n",
    "Output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Produces a prompt specific to the model\n",
    "def modelspecific_prompt(promptmessage):\n",
    "    # As per https://huggingface.co/TheBloke/phi-2-GGUF\n",
    "    return f\"Instruct: {promptmessage}\\nOutput:\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7. Service Context\n",
    "\n",
    "For chunking the document into tokens using the embedding model and our LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import Settings\n",
    "\n",
    "Settings.llm = llm\n",
    "Settings.embed_model = embed_model\n",
    "Settings.chunk_size=256 # Number of tokens in each chunk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8. Index documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "index = VectorStoreIndex.from_documents(documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 9. Query Engine\n",
    "\n",
    "Create a query engine, specifying how many citations we want to get back from the searched text (in this case 3).\n",
    "\n",
    "The DB_DOC_ID_KEY is used to get back the filename of the original document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.query_engine import CitationQueryEngine\n",
    "query_engine = CitationQueryEngine.from_args(\n",
    "    index,\n",
    "    similarity_top_k=3,\n",
    "    # here we can control how granular citation sources are, the default is 512\n",
    "    # citation_chunk_size=256 #256,\n",
    ")\n",
    "\n",
    "# For citations we get the document info\n",
    "DB_DOC_ID_KEY = \"db_document_id\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "service_context.node_parser:\n",
      "Chunk Overlap: 200\n",
      "Chunk Size: 256\n",
      "Paragraph Separator: \\n\\n\\n\n",
      "Secondary Chunking RegEx: [^,.;。？！]+[,.;。？！]?\n",
      "Include Metadata: True\n",
      "\n",
      "service_context.prompt_helper:\n",
      "Context Window: 2048\n",
      "Chunk Overlap Ratio: 0.1\n",
      "Chunk Size Limit: None\n",
      "Chunk Separator: '   '\n",
      "Num Output: ' 1024 '\n"
     ]
    }
   ],
   "source": [
    "# Let's look at the chunking and prompting parameters from our service_context\n",
    "\n",
    "print(\"service_context.node_parser:\")\n",
    "print(\"Chunk Overlap:\",Settings.node_parser.chunk_overlap)\n",
    "print(\"Chunk Size:\",Settings.node_parser.chunk_size)\n",
    "print(\"Paragraph Separator:\",Settings.node_parser.paragraph_separator.replace(\"\\n\",\"\\\\n\"))\n",
    "print(\"Secondary Chunking RegEx:\",Settings.node_parser.secondary_chunking_regex)\n",
    "print(\"Include Metadata:\",Settings.node_parser.include_metadata)\n",
    "\n",
    "print(\"\\nservice_context.prompt_helper:\")\n",
    "print(\"Context Window:\", Settings.prompt_helper.context_window) # The maximum context size that will get sent to the LLM.\n",
    "print(\"Chunk Overlap Ratio:\", Settings.prompt_helper.chunk_overlap_ratio) # The percentage token amount that each chunk should overlap.\n",
    "print(\"Chunk Size Limit:\", Settings.prompt_helper.chunk_size_limit) # The maximum size of a chunk.\n",
    "print(\"Chunk Separator: '\", Settings.prompt_helper.separator, \"'\") # The separator when chunking tokens.\n",
    "print(\"Num Output: '\", Settings.prompt_helper.num_output, \"'\") # The number of tokens that should be left to generate in the prompt (default is 256)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 10. Prompt and Response function\n",
    "\n",
    "Pass in a question, get a response back.\n",
    "\n",
    "IMPORTANT: The prompt is set here, adjust it to match what you want the LLM to act like and do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def RunQuestion(questionText):\n",
    "    # Excluding the system prompt as the model is including it (even a short version of it) is causing lack of responses in some cases and it is not consistently answering.\n",
    "    prompt = \"\" # \"You are a story teller who likes to elaborate and answers questions in a positive, helpful and interesting way, so please answer the following question - \"\n",
    "    \n",
    "    prompt = prompt + questionText\n",
    "\n",
    "    queryQuestion = modelspecific_prompt(prompt)\n",
    "\n",
    "    response = query_engine.query(queryQuestion)\n",
    "\n",
    "    return response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 11. Questions to test with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "TestQuestions = [\n",
    "    \"Summarise this story for me\",\n",
    "    \"Who was the main protagonist?\",\n",
    "    \"Did they have any children? If so, what were their names?\",\n",
    "    \"Did anything eventful happen?\",\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 12. Run Questions through model (this can take a while) and see citations\n",
    "\n",
    "Runs each test question, saves it to a dictionary for output in the last step.\n",
    "\n",
    "Note: Citations are the source documents used and the text the response is based on. This is important for RAG so you can reference these documents for the user, and to ensure it's utilising the right documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "1/4: Summarise this story for me\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =      66.99 ms\n",
      "llama_print_timings:      sample time =      81.91 ms /   195 runs   (    0.42 ms per token,  2380.55 tokens per second)\n",
      "llama_print_timings: prompt eval time =     150.01 ms /  1008 tokens (    0.15 ms per token,  6719.60 tokens per second)\n",
      "llama_print_timings:        eval time =    2133.92 ms /   194 runs   (   11.00 ms per token,    90.91 tokens per second)\n",
      "llama_print_timings:       total time =    2891.29 ms /  1202 tokens\n",
      "Llama.generate: prefix-match hit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source 1 of 3: |Thundertooth Part 1.docx| Source 1:\n",
      "Thundertooth faced a dilemma – he was hungry, but he couldn't bring himself to feast on the humans who scurried around like ants. As his hunger grew, he stumbled upon a park, an oasis of greenery amidst the concrete and steel. The park was adorned with holographic flowers that emitted a gentle glow, creating an ethereal atmosphere.\n",
      "\n",
      "\n",
      "\n",
      "While Thundertooth marveled at the beauty of the park, the mayor of the city happened to be passing by. Mayor Eleanor Grace, a charismatic and forward-thinking leader, was immediately intrigued by the sight of the talking dinosaur. She approached Thundertooth with a mix of curiosity and caution.\n",
      "\n",
      "\n",
      "\n",
      "\"Hello there, majestic creature. What brings you to our time?\" Mayor Grace inquired, her voice calm and reassuring.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth, though initially startled, found comfort in the mayor's soothing tone. In broken sentences, he explained his journey through time, the strange portal, and his hunger dilemma. Mayor Grace listened intently, her eyes widening with amazement at the tale of the prehistoric dinosaur navigating the future.\n",
      "\n",
      "Source 2 of 3: |Thundertooth Part 3.docx| Source 2:\n",
      "2. **Echo**: With his extraordinary mimicry abilities, Echo would amplify the emergency signals, ensuring that every citizen received timely warnings and instructions for evacuation.\n",
      "\n",
      "\n",
      "\n",
      "3. **Sapphire**: Harnessing her calming and healing powers, Sapphire would assist in calming the panicked masses, ensuring an orderly and efficient evacuation.\n",
      "\n",
      "\n",
      "\n",
      "4. **Ignis**: Drawing upon his fiery talents, Ignis would create controlled bursts of heat, attempting to alter the meteor's trajectory and reduce its destructive force.\n",
      "\n",
      "\n",
      "\n",
      "As the citizens evacuated to designated shelters, the Thundertooth family sprang into action. Lumina worked tirelessly to strengthen the city's energy systems, Echo echoed evacuation orders through the city's speakers, Sapphire offered comfort to those in distress, and Ignis unleashed controlled bursts of flames towards the approaching meteor.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth stood at the forefront, using his mighty roar to coordinate and inspire the efforts of the city's inhabitants. The ground trembled as the meteor drew closer, but the Thundertooth family's coordinated efforts began to take effect. Lumina's force field shimmered to life, deflecting the meteor's deadly path. Echo's amplified warnings reached every corner of the city, ensuring that no one was left behind.\n",
      "\n",
      "Source 3 of 3: |Thundertooth Part 2.docx| Source 3:\n",
      "Thundertooth\n",
      "\n",
      "\n",
      "\n",
      "Embraced by the futuristic city and its inhabitants, Thundertooth found a sense of purpose beyond merely satisfying his hunger. Inspired by the advanced technology surrounding him, he decided to channel his creativity into something extraordinary. With the help of the city's brilliant engineers, Thundertooth founded a one-of-a-kind toy factory that produced amazing widgets – magical, interactive toys that captivated the hearts of both children and adults alike.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth's toy factory became a sensation, and its creations were highly sought after. The widgets incorporated cutting-edge holographic displays, levitation technology, and even the ability to change shapes and colors with a mere thought. Children across the city rejoiced as they played with these incredible toys that seemed to bring their wildest fantasies to life.\n",
      "\n",
      "\n",
      "\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "\n",
      "2/4: Who was the main protagonist?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =      66.99 ms\n",
      "llama_print_timings:      sample time =      35.07 ms /    84 runs   (    0.42 ms per token,  2395.00 tokens per second)\n",
      "llama_print_timings: prompt eval time =     137.19 ms /   824 tokens (    0.17 ms per token,  6006.40 tokens per second)\n",
      "llama_print_timings:        eval time =     914.64 ms /    83 runs   (   11.02 ms per token,    90.75 tokens per second)\n",
      "llama_print_timings:       total time =    1306.00 ms /   907 tokens\n",
      "Llama.generate: prefix-match hit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source 1 of 3: |Thundertooth Part 3.docx| Source 1:\n",
      "2. **Echo**: With his extraordinary mimicry abilities, Echo would amplify the emergency signals, ensuring that every citizen received timely warnings and instructions for evacuation.\n",
      "\n",
      "\n",
      "\n",
      "3. **Sapphire**: Harnessing her calming and healing powers, Sapphire would assist in calming the panicked masses, ensuring an orderly and efficient evacuation.\n",
      "\n",
      "\n",
      "\n",
      "4. **Ignis**: Drawing upon his fiery talents, Ignis would create controlled bursts of heat, attempting to alter the meteor's trajectory and reduce its destructive force.\n",
      "\n",
      "\n",
      "\n",
      "As the citizens evacuated to designated shelters, the Thundertooth family sprang into action. Lumina worked tirelessly to strengthen the city's energy systems, Echo echoed evacuation orders through the city's speakers, Sapphire offered comfort to those in distress, and Ignis unleashed controlled bursts of flames towards the approaching meteor.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth stood at the forefront, using his mighty roar to coordinate and inspire the efforts of the city's inhabitants. The ground trembled as the meteor drew closer, but the Thundertooth family's coordinated efforts began to take effect. Lumina's force field shimmered to life, deflecting the meteor's deadly path. Echo's amplified warnings reached every corner of the city, ensuring that no one was left behind.\n",
      "\n",
      "Source 2 of 3: |Thundertooth Part 3.docx| Source 2:\n",
      "Thundertooth nodded, understanding the gravity of the situation. He gathered Lumina, Echo, Sapphire, and Ignis, explaining the urgency and the role each of them would play in the impending crisis.\n",
      "\n",
      "\n",
      "\n",
      "1. **Lumina**: Utilizing her deep understanding of technology, Lumina would enhance the city's energy systems to generate a powerful force field, providing a protective barrier against the meteor's impact.\n",
      "\n",
      "\n",
      "\n",
      "2. **Echo**: With his extraordinary mimicry abilities, Echo would amplify the emergency signals, ensuring that every citizen received timely warnings and instructions for evacuation.\n",
      "\n",
      "\n",
      "\n",
      "3. **Sapphire**: Harnessing her calming and healing powers, Sapphire would assist in calming the panicked masses, ensuring an orderly and efficient evacuation.\n",
      "\n",
      "\n",
      "\n",
      "4. **Ignis**: Drawing upon his fiery talents, Ignis would create controlled bursts of heat, attempting to alter the meteor's trajectory and reduce its destructive force.\n",
      "\n",
      "\n",
      "\n",
      "As the citizens evacuated to designated shelters, the Thundertooth family sprang into action. Lumina worked tirelessly to strengthen the city's energy systems, Echo echoed evacuation orders through the city's speakers, Sapphire offered comfort to those in distress, and Ignis unleashed controlled bursts of flames towards the approaching meteor.\n",
      "\n",
      "Source 3 of 3: |Thundertooth Part 2.docx| Source 3:\n",
      "Thundertooth\n",
      "\n",
      "\n",
      "\n",
      "Embraced by the futuristic city and its inhabitants, Thundertooth found a sense of purpose beyond merely satisfying his hunger. Inspired by the advanced technology surrounding him, he decided to channel his creativity into something extraordinary. With the help of the city's brilliant engineers, Thundertooth founded a one-of-a-kind toy factory that produced amazing widgets – magical, interactive toys that captivated the hearts of both children and adults alike.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth's toy factory became a sensation, and its creations were highly sought after. The widgets incorporated cutting-edge holographic displays, levitation technology, and even the ability to change shapes and colors with a mere thought. Children across the city rejoiced as they played with these incredible toys that seemed to bring their wildest fantasies to life.\n",
      "\n",
      "\n",
      "\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "\n",
      "3/4: Did they have any children? If so, what were their names?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =      66.99 ms\n",
      "llama_print_timings:      sample time =      14.21 ms /    34 runs   (    0.42 ms per token,  2393.52 tokens per second)\n",
      "llama_print_timings: prompt eval time =     120.24 ms /   758 tokens (    0.16 ms per token,  6303.90 tokens per second)\n",
      "llama_print_timings:        eval time =     362.38 ms /    33 runs   (   10.98 ms per token,    91.07 tokens per second)\n",
      "llama_print_timings:       total time =     587.09 ms /   791 tokens\n",
      "Llama.generate: prefix-match hit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source 1 of 3: |Thundertooth Part 2.docx| Source 1:\n",
      "Thundertooth's toy factory became a sensation, and its creations were highly sought after. The widgets incorporated cutting-edge holographic displays, levitation technology, and even the ability to change shapes and colors with a mere thought. Children across the city rejoiced as they played with these incredible toys that seemed to bring their wildest fantasies to life.\n",
      "\n",
      "\n",
      "\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "\n",
      "\n",
      "Lumina: The eldest of Thundertooth's children, Lumina inherited her mother's intelligence and her father's sense of wonder. With sparkling scales that emitted a soft glow, Lumina had the ability to generate light at will. She became fascinated with technology, often spending hours tinkering with gadgets and inventing new ways to enhance the widgets produced in the family's factory.\n",
      "\n",
      "Source 2 of 3: |Thundertooth Part 2.docx| Source 2:\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "\n",
      "\n",
      "Lumina: The eldest of Thundertooth's children, Lumina inherited her mother's intelligence and her father's sense of wonder. With sparkling scales that emitted a soft glow, Lumina had the ability to generate light at will. She became fascinated with technology, often spending hours tinkering with gadgets and inventing new ways to enhance the widgets produced in the family's factory.\n",
      "\n",
      "\n",
      "\n",
      "Echo: The second-born, Echo, had a gift for mimicry. He could perfectly replicate any sound or voice he heard, providing entertainment to the entire city. His playful nature and ability to bring joy to those around him made him a favorite among the neighborhood children.\n",
      "\n",
      "Source 3 of 3: |Thundertooth Part 2.docx| Source 3:\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "\n",
      "\n",
      "Lumina: The eldest of Thundertooth's children, Lumina inherited her mother's intelligence and her father's sense of wonder. With sparkling scales that emitted a soft glow, Lumina had the ability to generate light at will. She became fascinated with technology, often spending hours tinkering with gadgets and inventing new ways to enhance the widgets produced in the family's factory.\n",
      "\n",
      "\n",
      "\n",
      "Echo: The second-born, Echo, had a gift for mimicry. He could perfectly replicate any sound or voice he heard, providing entertainment to the entire city. His playful nature and ability to bring joy to those around him made him a favorite among the neighborhood children.\n",
      "\n",
      "\n",
      "\n",
      "Sapphire: Sapphire, the third sibling, had scales that shimmered like precious gems. She possessed a unique talent for calming and healing, a trait she inherited from both her parents. Whenever someone in the city felt stressed or unwell, Sapphire would extend her gentle touch, bringing comfort and tranquility.\n",
      "\n",
      "\n",
      "4/4: Did anything eventful happen?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =      66.99 ms\n",
      "llama_print_timings:      sample time =     432.67 ms /  1024 runs   (    0.42 ms per token,  2366.69 tokens per second)\n",
      "llama_print_timings: prompt eval time =     128.63 ms /   806 tokens (    0.16 ms per token,  6266.03 tokens per second)\n",
      "llama_print_timings:        eval time =   11790.75 ms /  1023 runs   (   11.53 ms per token,    86.76 tokens per second)\n",
      "llama_print_timings:       total time =   15739.99 ms /  1829 tokens\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source 1 of 3: |Thundertooth Part 3.docx| Source 1:\n",
      "As Ignis's controlled bursts of flames interacted with the meteor, it began to change course. The combined efforts of the Thundertooth family, guided by their unique talents, diverted the catastrophic collision. The meteor, once destined for destruction, now harmlessly sailed past the Earth, leaving the city and its inhabitants unscathed.\n",
      "\n",
      "\n",
      "\n",
      "The citizens, emerging from their shelters, erupted into cheers of gratitude. Mayor Grace approached Thundertooth, expressing her heartfelt thanks for the family's heroic efforts. The Thundertooth family, tired but triumphant, basked in the relief of having saved their beloved city from imminent disaster.\n",
      "\n",
      "\n",
      "\n",
      "In the wake of the crisis, the citizens of the futuristic city hailed Thundertooth and his family as true heroes. The toy factory that once brought joy to children now became a symbol of resilience and unity. The Thundertooth family's legacy was forever etched in the city's history, a testament to the power of cooperation and the extraordinary capabilities that could emerge when dinosaurs and humans worked together for the greater good.\n",
      "\n",
      "Source 2 of 3: |Thundertooth Part 3.docx| Source 2:\n",
      "Thundertooth nodded, understanding the gravity of the situation. He gathered Lumina, Echo, Sapphire, and Ignis, explaining the urgency and the role each of them would play in the impending crisis.\n",
      "\n",
      "\n",
      "\n",
      "1. **Lumina**: Utilizing her deep understanding of technology, Lumina would enhance the city's energy systems to generate a powerful force field, providing a protective barrier against the meteor's impact.\n",
      "\n",
      "\n",
      "\n",
      "2. **Echo**: With his extraordinary mimicry abilities, Echo would amplify the emergency signals, ensuring that every citizen received timely warnings and instructions for evacuation.\n",
      "\n",
      "\n",
      "\n",
      "3. **Sapphire**: Harnessing her calming and healing powers, Sapphire would assist in calming the panicked masses, ensuring an orderly and efficient evacuation.\n",
      "\n",
      "\n",
      "\n",
      "4. **Ignis**: Drawing upon his fiery talents, Ignis would create controlled bursts of heat, attempting to alter the meteor's trajectory and reduce its destructive force.\n",
      "\n",
      "\n",
      "\n",
      "As the citizens evacuated to designated shelters, the Thundertooth family sprang into action. Lumina worked tirelessly to strengthen the city's energy systems, Echo echoed evacuation orders through the city's speakers, Sapphire offered comfort to those in distress, and Ignis unleashed controlled bursts of flames towards the approaching meteor.\n",
      "\n",
      "Source 3 of 3: |Thundertooth Part 3.docx| Source 3:\n",
      "2. **Echo**: With his extraordinary mimicry abilities, Echo would amplify the emergency signals, ensuring that every citizen received timely warnings and instructions for evacuation.\n",
      "\n",
      "\n",
      "\n",
      "3. **Sapphire**: Harnessing her calming and healing powers, Sapphire would assist in calming the panicked masses, ensuring an orderly and efficient evacuation.\n",
      "\n",
      "\n",
      "\n",
      "4. **Ignis**: Drawing upon his fiery talents, Ignis would create controlled bursts of heat, attempting to alter the meteor's trajectory and reduce its destructive force.\n",
      "\n",
      "\n",
      "\n",
      "As the citizens evacuated to designated shelters, the Thundertooth family sprang into action. Lumina worked tirelessly to strengthen the city's energy systems, Echo echoed evacuation orders through the city's speakers, Sapphire offered comfort to those in distress, and Ignis unleashed controlled bursts of flames towards the approaching meteor.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth stood at the forefront, using his mighty roar to coordinate and inspire the efforts of the city's inhabitants. The ground trembled as the meteor drew closer, but the Thundertooth family's coordinated efforts began to take effect. Lumina's force field shimmered to life, deflecting the meteor's deadly path. Echo's amplified warnings reached every corner of the city, ensuring that no one was left behind.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "qa_pairs = []\n",
    "\n",
    "for indexNum, question in enumerate(TestQuestions, start=1):\n",
    "    question = question.strip() # Clean up\n",
    "\n",
    "    print(f\"\\n{indexNum}/{len(TestQuestions)}: {question}\")\n",
    "\n",
    "    response = RunQuestion(question) # Query and get  response\n",
    "\n",
    "    qa_pairs.append((question.strip(), str(response).strip())) # Add to our output array\n",
    "\n",
    "    # Displays the citations\n",
    "    for indexNumCit, node in enumerate(response.source_nodes, start=1):\n",
    "        print(f\"Source {indexNumCit} of {len(response.source_nodes)}: |{node.node.metadata['file_name']}| {node.node.get_text()}\")\n",
    "\n",
    "    # Uncomment the following line if you want to test just the first question\n",
    "    # break "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 13. Output responses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/4 Summarise this story for me\n",
      "\n",
      "The story is about Thundertooth, a talking dinosaur who finds himself in the future city of Thunderville. He faces a dilemma of hunger but is intrigued by the futuristic city's beauty. The mayor of Thunderville, Eleanor Grace, approaches Thundertooth with curiosity and caution. Thundertooth explains his journey through time and his hunger dilemma to the mayor. The mayor is amazed by Thundertooth's tale and offers her assistance. Thundertooth's family, consisting of Lumina, Echo, Sapphire, Ignis, and their children, work together to protect the city from an approaching meteor. Thundertooth finds purpose in the futuristic city and starts a toy factory that produces amazing widgets. The widgets become highly sought after and bring joy to children across the city. Thundertooth meets Seraphina and starts a family with four children who possess unique characteristics. The story ends with Thundertooth's heartwarming family life in the modern world. [/INST]\n",
      "\n",
      "--------\n",
      "\n",
      "2/4 Who was the main protagonist?\n",
      "\n",
      "The main protagonist of the story is Thundertooth. He is the central character who embarks on a journey to save his city from the approaching meteor. Throughout the story, Thundertooth demonstrates bravery, resourcefulness, and leadership qualities as he works alongside his family to protect their home and its inhabitants. His determination to use his unique abilities for the greater good makes him the main protagonist of the story.\n",
      "\n",
      "--------\n",
      "\n",
      "3/4 Did they have any children? If so, what were their names?\n",
      "\n",
      "Yes, Thundertooth and Seraphina had four children: Lumina, Echo, Sapphire, and the fourth child's name is unknown. [/INST]\n",
      "\n",
      "--------\n",
      "\n",
      "4/4 Did anything eventful happen?\n",
      "\n",
      "[1] The citizens emerged from their shelters, cheering in gratitude for the Thundertooth family's heroic efforts in saving the city from imminent disaster.\n",
      "[2] The Thundertooth family was hailed as true heroes by the citizens of the futuristic city.\n",
      "[3] The Thundertooth family's legacy became forever etched in the city's history, symbolizing resilience and unity in the face of adversity.\n",
      "[4] The Thundertooth family's coordinated efforts successfully diverted the meteor's destructive path, saving the city from catastrophe.\n",
      "[5] The Thundertooth family's heroic actions brought the citizens together, strengthening the bond between dinosaurs and humans in the city.\n",
      "[6] The Thundertooth family's legacy served as a reminder of the extraordinary capabilities that can emerge when different species work together for the greater good.\n",
      "[7] The Thundertooth family's heroic efforts inspired future generations to embrace cooperation and unity in the face of challenges.\n",
      "[8] The Thundertooth family's legacy became a symbol of hope and resilience in the city, reminding its inhabitants of the power of collective action.\n",
      "[9] The Thundertooth family's heroic actions brought about a renewed sense of pride and unity among the citizens of the futuristic city.\n",
      "[10] The Thundertooth family's legacy became a testament to the extraordinary capabilities that can emerge when dinosaurs and humans work together for the greater good.\n",
      "[11] The Thundertooth family's heroic actions brought about a renewed sense of hope and unity among the citizens of the futuristic city.\n",
      "[12] The Thundertooth family's legacy became a symbol of hope and resilience in the city, reminding its inhabitants of the power of collective action.\n",
      "[13] The Thundertooth family's heroic actions inspired future generations to embrace cooperation and unity in the face of challenges.\n",
      "[14] The Thundertooth family's legacy became a symbol of hope and resilience in the city, reminding its inhabitants of the power of collective action.\n",
      "[15] The Thundertooth family's heroic actions brought about a renewed sense of pride and unity among the citizens of the futuristic city.\n",
      "[16] The Thundertooth family's legacy became a testament to the extraordinary capabilities that can emerge when dinosaurs and humans work together for the greater good.\n",
      "[17] The Thundertooth family's heroic actions inspired future generations to embrace cooperation and unity in the face of challenges.\n",
      "[18] The Thundertooth family's legacy became a symbol of hope and resilience in the city, reminding its inhabitants of the power of collective action.\n",
      "[19] The Thundertooth family's heroic actions brought about a renewed sense of pride and unity among the citizens of the futuristic city.\n",
      "[20] The Thundertooth family's legacy became a testament to the extraordinary capabilities that can emerge when dinosaurs and humans work together for the greater good.\n",
      "[21] The Thundertooth family's heroic actions inspired future generations to embrace cooperation and unity in the face of challenges.\n",
      "[22] The Thundertooth family's legacy became a symbol of hope and resilience in the city, reminding its inhabitants of the power of collective action.\n",
      "[23] The Thundertooth family's heroic actions brought about a renewed sense of pride and unity among the citizens of the futuristic city.\n",
      "[24] The Thundertooth family's legacy became a testament to the extraordinary capabilities that can emerge when dinosaurs and humans work together for the greater good.\n",
      "[25] The Thundertooth family's heroic actions inspired future generations to embrace cooperation and unity in the face of challenges.\n",
      "[26] The Thundertooth family's legacy became a symbol of hope and resilience in the city, reminding its inhabitants of the power of collective action.\n",
      "[27] The Thundertooth family's heroic actions brought about a renewed sense of pride and unity among the citizens of the futuristic city.\n",
      "[28] The Thundertooth family's legacy became a testament to the extraordinary capabilities that can emerge when dinosaurs and humans work together for the greater good.\n",
      "[29] The Thundertooth family's heroic actions inspired future generations to embrace cooperation and unity in the face of challenges.\n",
      "[30] The Thundertooth family's legacy became a symbol of hope and resilience in the city, reminding its inhabitants of the power of collective action.\n",
      "[31] The Thundertooth family's heroic actions brought about a renewed sense of pride and unity among the citizens of the futuristic city.\n",
      "[32] The Thundertooth family's legacy became a testament to the extraordinary capabilities that can emerge when dinosaurs and humans work together for the greater good.\n",
      "[33] The Thundertooth family's heroic actions inspired future generations to embrace cooperation and unity in the face of challenges.\n",
      "[34] The Thundertooth family's legacy became a symbol of hope and resilience in the city, reminding its inhabitants of the power of collective action.\n",
      "[35] The Thundertooth family's heroic actions brought about a renewed sense of pride and unity\n",
      "\n",
      "--------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for indexQA, (question, answer) in enumerate(qa_pairs, start=1):\n",
    "    print(f\"{indexQA}/{len(qa_pairs)} {question}\\n\\n{answer}\\n\\n--------\\n\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llamaindexgeneric",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}