tutorials/synthetic-data-hello-world/Synthetic Data Generation - Hello World Examples.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started with Synthetic Data Generation Powered by NeMo Curator\n",
    "\n",
    "In the following notebook, we'll be exploring all of the amazing out-of-the-box functionality of the NeMo Curator Synthetic Data Generation (SDG) tooling.\n",
    "\n",
    "First, we'll work through an example of a pipeline in a piece-wise fashion - spending time exploring exactly how flexible NeMo Curator's SDG functionality is. Then, we'll explore all of the built-in pipelines for generating synthetic data for a number of different tasks. \n",
    "\n",
    "In order to get started, though, we'll need to install NeMo Curator! \n",
    "\n",
    "> NOTE: Please ensure you meet the [requirements](https://github.com/NVIDIA/NeMo-Curator/tree/main?tab=readme-ov-file#install-nemo-curator) before proceeding!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing NeMo Curator Dependencies\n",
    "\n",
    "We'll install NeMo Curator from source! First, let's `git clone` the repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'NeMo-Curator'...\n",
      "remote: Enumerating objects: 2051, done.\u001b[K\n",
      "remote: Counting objects: 100% (1512/1512), done.\u001b[K\n",
      "remote: Compressing objects: 100% (837/837), done.\u001b[K\n",
      "remote: Total 2051 (delta 983), reused 1002 (delta 666), pack-reused 539 (from 1)\u001b[K\n",
      "Receiving objects: 100% (2051/2051), 2.28 MiB | 15.29 MiB/s, done.\n",
      "Resolving deltas: 100% (1236/1236), done.\n",
      "/home/chris/Code/NVIDIA/NeMo-Curator/tutorials/synthetic-data-hello-world/NeMo-Curator\n"
     ]
    }
   ],
   "source": [
    "!git clone https://github.com/NVIDIA/NeMo-Curator.git\n",
    "%cd NeMo-Curator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can install the required libraries!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -qU wheel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -qU ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the NeMo Curator OpenAI Client\n",
    "\n",
    "To ensure compatibility within the NeMo Curator SDG tooling, we're going to use a specialized OpenAI Client. This is based on the [OpenAI Python API](https://github.com/openai/openai-python?tab=readme-ov-file#openai-python-api-library) library - but with a few modifications to allow seamless use for generating Synthetic Data Generation.\n",
    "\n",
    "> NOTE: While we're going to be relying on the `build.nvidia.com` API endpoints for this example notebook, you can use this same flow with a model deployed as an NVIDIA NIM for LLMs which can be found [here](https://github.com/NVIDIA/NeMo-Curator/blob/main/docs/user-guide/syntheticdata.rst#connecting-to-an-llm-service).\n",
    "\n",
    "You'll need to make sure you have a NVIDIA API key - which you can obtain by following this process: \n",
    "\n",
    "1. Login (or sign up) through [build.nvidia.com](https://build.nvidia.com/explore/discover).\n",
    "2. Click the `Get API Key` button available on the the `nvidia/nemotron-4-340b-instruct` page, found [here](https://build.nvidia.com/nvidia/nemotron-4-340b-instruct)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import getpass\n",
    "\n",
    "os.environ[\"NVIDIA_API_KEY\"] = getpass.getpass(\"Please provide your API Catalogue NVIDIA API Key:\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we're going to want to initialize the base OpenAI client. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "\n",
    "openai_client = OpenAI(\n",
    "    base_url=\"https://integrate.api.nvidia.com/v1\",\n",
    "    api_key=os.environ[\"NVIDIA_API_KEY\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can initalize our NeMo Curator `OpenAIClient`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/chris/anaconda3/envs/curator_fresh/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "from nemo_curator import OpenAIClient\n",
    "\n",
    "curator_openai_client = OpenAIClient(openai_client)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chat Model Usage\n",
    "\n",
    "Now we can look at how to use our NeMo Curator `OpenAIClient` to generate a response. \n",
    "\n",
    "As you can see - the structure of the request is very close to the traditional OpenAI client!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In the realm of computing, where data's the king,\n",
      "GPU power makes everything sing.\n",
      "Parallel processing, so neat,\n",
      "Makes complex tasks a treat,\n",
      "A wonder of tech, it's truly a thing!\n",
      "\n",
      "With thousands of cores, in silicon etched,\n",
      "Through machine learning, they're well-matched.\n",
      "Nvidia, AMD, in the race,\n",
      "To accelerate every place,\n",
      "GPU computing, a marvel, is hatched.\n",
      "\n",
      "From gaming to AI, and scientific research,\n",
      "GPUs help us leap, not just lurch.\n",
      "So here's to the engineers, so bright,\n",
      "Who brought us this marvel, pure delight,\n",
      "GPU computing, a true gem, we search!\n",
      "\n"
     ]
    }
   ],
   "source": [
    "responses = curator_openai_client.query_model(\n",
    "    model=\"nvidia/nemotron-4-340b-instruct\",\n",
    "    messages=[\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"Write a limerick about the wonders of GPU computing.\",\n",
    "        }\n",
    "    ],\n",
    "    temperature=0.2,\n",
    "    top_p=0.7,\n",
    "    max_tokens=1024,\n",
    ")\n",
    "print(responses[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reward Model Usage\n",
    "\n",
    "We can use the same client to query NVIDIA's [best-in-class](https://huggingface.co/spaces/allenai/reward-bench) Reward Model - [Nemotron-4 340B Reward](https://build.nvidia.com/nvidia/nemotron-4-340b-reward) using the [`query_reward_model`](https://github.com/NVIDIA/NeMo-Curator/blob/cd4c4907bd4d87cd11d0f37be4ae0fe167a79696/nemo_curator/services/openai_client.py#L137) method of our `OpenAIClient`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = \"nvidia/nemotron-4-340b-reward\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `query_reward_model` method expects a conversation between a User and an Assistant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\n",
    "        \"role\": \"user\", \n",
    "        \"content\": \"I am going to Paris, what should I see?\"\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": \"Ah, Paris, the City of Light! There are so many amazing things to see and do in this beautiful city...\",\n",
    "    },\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After that, we can simply fire off our request!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'helpfulness': 1.4765625, 'correctness': 1.6171875, 'coherence': 3.21875, 'complexity': 0.640625, 'verbosity': 0.365234375}\n"
     ]
    }
   ],
   "source": [
    "rewards = curator_openai_client.query_reward_model(messages=messages, model=model)\n",
    "print(rewards)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Nemotron-4 340B Reward model will provide us the scores (between 0 and 4) for each of the 5 SteerLM attributes:\n",
    "\n",
    "- **Helpfulness:** Overall helpfulness of the response to the prompt.\n",
    "- **Correctness:** Inclusion of all pertinent facts without errors.\n",
    "- **Coherence:** Consistency and clarity of expression.\n",
    "- **Complexity:** Intellectual depth required to write response (i.e. whether the response can be written by anyone with basic language competency or requires deep domain expertise).\n",
    "- **Verbosity:** Amount of detail included in the response, relative to what is asked for in the prompt.\n",
    "\n",
    "These can be used as a filter for any of the individual attributes, or utilized to verify specific attributes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using The `NemotronGenerator`\n",
    "\n",
    "The NeMo Curator Synthetic Data Generation (SDG) features are primarily accessed through the `NemotronGenerator` class. \n",
    "\n",
    "This useful wrapped helps expose both: \n",
    "\n",
    "1. Pre-built SDG pipelines \n",
    "2. A number of specific generation utilities, which we'll explore in the following section of the notebook.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nemo_curator.synthetic import NemotronGenerator\n",
    "\n",
    "generator = NemotronGenerator(curator_openai_client)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you'd like to skip forward to a specific pipeline, you can find them here:\n",
    "\n",
    "- [Math Question Generation Pipeline](#math-question-generation-pipeline)\n",
    "- [Writing Task Generation Pipeline](#writing-task-generation-pipeline)\n",
    "- [Open Question Generation Pipeline](#open-question-pipeline)\n",
    "- [Closed Question Generation Pipeline](#closed-question-pipeline)\n",
    "- [Python Question Generation Pipeline](#python-question-generation-pipeline)\n",
    "- [Dialogue Generation Pipeline](#dialogue-generation-pipeline)\n",
    "- [Two-Turn Prompt Generation Pipeline](#two-turn-prompt-generation-pipeline)\n",
    "- [Entity Classification](#entity-classification)\n",
    "    - [Classify Math Entity](#classify-math-entity)\n",
    "    - [Classify Python Entity](#classify-python-entity)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploring the Math Question Generation Pipeline\n",
    "\n",
    "Before heading into the pre-built pipelines, we're going to \"break-apart\" an existing pipeline, in this case: the Math Question Generation Pipeline - and see the granular customization that Nemo Curator provides for each step. \n",
    "\n",
    "We're going to work through the following process, which is detailed in the [Nemotron-4 340B Technical Report](https://arxiv.org/pdf/2406.11704):\n",
    "\n",
    "1. Generate `n` Macro Topics - Have our LLM generate `n` broad topics relating to daily life, the world, etc.\n",
    "2. Generate `n` Sub Topics - Have our LLM take each Macro Topic and generate `n` topics relating to the Macro Topics.\n",
    "3. Generate `n` Questions - Have our LLM take each subtopic and generate `n` questions related to that topic (at the desired level)\n",
    "\n",
    "Let's dive in!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model Selection and Configs\n",
    "\n",
    "First, we'll emulate the process as outlined by the Nemotron-4 340B Technical Report by selecting the `Mixtral-8x7B-Instruct-v0.1` model, as well as some reasonable generation parameters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = \"mistralai/mixtral-8x7b-instruct-v0.1\"\n",
    "model_kwargs = {\n",
    "    \"temperature\": 0.1,\n",
    "    \"top_p\": 0.9,\n",
    "    \"max_tokens\": 1024,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Generating `n` Macro Topics\n",
    "\n",
    "Our first step is to generate our Macro Topics. \n",
    "\n",
    "Let's look at the prompt that drives this process as well, to get a better understanding of what's happening \"under the hood\":\n",
    "\n",
    "```python\n",
    "\"Can you generate {n_macro_topics} comprehensive topics that encompass various aspects of our daily life, the world, and science? Your answer should be a list of topics. Make the topics as diverse as possible.For example, 1. Food and drinks. \\n2. Technology.\\n\"\n",
    "```\n",
    "\n",
    "To do this, we'll use the [`generate_macro_topics`](https://github.com/NVIDIA/NeMo-Curator/blob/cd4c4907bd4d87cd11d0f37be4ae0fe167a79696/nemo_curator/synthetic/nemotron.py#L115) method of our `NemotronGenerator`.\n",
    "\n",
    "> NOTE: All prompt templates are fully customizable, and we'll take a look at how we can do that in the upcoming cells!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the number of macro topics to generate\n",
    "n_macro_topics = 20\n",
    "\n",
    "# generate macro topics\n",
    "responses = generator.generate_macro_topics(\n",
    "    n_macro_topics=n_macro_topics, \n",
    "    model=model, \n",
    "    model_kwargs=model_kwargs\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at our response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Climate Change and Environmental Impact\n",
      "2. Mental Health and Well-being\n",
      "3. Space Exploration and Astronomy\n",
      "4. Global Health and Pandemics\n",
      "5. Renewable Energy and Sustainable Living\n",
      "6. Artificial Intelligence and Machine Learning\n",
      "7. Biodiversity and Conservation\n",
      "8. Virtual Reality and Gaming\n",
      "9. Nutrition and Diet\n",
      "10. Social Media and Online Communication\n",
      "11. Genetics and Genetic Engineering\n",
      "12. E-commerce and Online Shopping\n",
      "13. Neuroscience and Human Behavior\n",
      "14. Disaster Preparedness and Response\n",
      "15. Quantum Computing and Cryptography\n",
      "16. Education and Lifelong Learning\n",
      "17. Cybersecurity and Data Privacy\n",
      "18. Biotechnology and Synthetic Biology\n",
      "19. Transportation and Urban Planning\n",
      "20. Human Rights and Social Justice.\n"
     ]
    }
   ],
   "source": [
    "print(responses[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While this is a great start - we'd love to have this response in a Python list. \n",
    "\n",
    "Luckily for us, Nemo Curator has just the tool!\n",
    "\n",
    "We'll use the [`convert_response_to_yaml_list`](https://github.com/NVIDIA/NeMo-Curator/blob/cd4c4907bd4d87cd11d0f37be4ae0fe167a79696/nemo_curator/synthetic/nemotron.py#L61) method to accomplish this goal. \n",
    "\n",
    "> NOTE: Currently, this method is quite strict - and so custom parsing might be required depending on model choice, and use-case. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nemo_curator.synthetic.error import YamlConversionError\n",
    "\n",
    "while True:\n",
    "    try:\n",
    "        topic_list = generator.convert_response_to_yaml_list(\n",
    "            responses[0], model=model, model_kwargs=model_kwargs\n",
    "        )\n",
    "        break\n",
    "    except YamlConversionError as e:\n",
    "        print(f\"Hit: {e}, Retrying...\")\n",
    "        responses = generator.generate_macro_topics(\n",
    "            n_macro_topics=n_macro_topics, \n",
    "            model=model, \n",
    "            model_kwargs=model_kwargs\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now our response has been converted into a Python list - which is perfect for our next step: Generating subtopics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Climate Change and Environmental Impact\n"
     ]
    }
   ],
   "source": [
    "print(topic_list[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Generating `n` subtopics\n",
    "\n",
    "We'll proceed through the same process as we did above, but this time our prompt will reflect our desire to generate subtopics. \n",
    "\n",
    "Let's check it out:\n",
    "\n",
    "```python\n",
    "\"Can you generate {n_subtopics} comprehensive topics that encompass various aspects of {macro_topic}? Your answer should be a list of topics. Make the topics as diverse as possible.\"\n",
    "```\n",
    "\n",
    "Otherwise, we will use the [`generate_subtopics`](https://github.com/NVIDIA/NeMo-Curator/blob/cd4c4907bd4d87cd11d0f37be4ae0fe167a79696/nemo_curator/synthetic/nemotron.py#L147) method to fire off this subtask. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# number of subtopics to generate\n",
    "n_subtopics = 5\n",
    "\n",
    "# generate subtopics\n",
    "subtopic_responses = generator.generate_subtopics(\n",
    "    macro_topic=topic_list[0], n_subtopics=n_subtopics, model=model\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, if we look at our results - they are great, but they are not in a desirable format to integrate cleanly into a pipeline. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. \"Global Warming and the Role of Greenhouse Gases\": This topic could cover the science behind global warming, the impact of human activities (such as burning fossil fuels) on greenhouse gas emissions, and potential solutions to reduce our carbon footprint.\n",
      "\n",
      "2. \"Impact of Deforestation on Biodiversity and Climate Change\": This topic could explore the importance of forests in maintaining the planet's biodiversity, their role in carbon sequestration, and the devastating effects of deforestation on both.\n",
      "\n",
      "3. \"Climate Change and Ocean Acidification\": This topic could delve into how increased carbon dioxide levels in the atmosphere are leading to ocean acidification, its impact on marine life, and potential consequences for the food chain and human societies.\n",
      "\n",
      "4. \"Renewable Energy Sources and Sustainable Future\": This topic could examine various types of renewable energy (solar, wind, hydro, etc.), their advantages and challenges, and how they can help mitigate climate change while ensuring a sustainable future.\n",
      "\n",
      "5. \"Climate Change Mitigation and Adaptation Strategies\": This topic could discuss different strategies to combat climate change, ranging from reducing emissions (mitigation) to coping with its impacts (adaptation). It could also look at international agreements like the Paris Agreement and national policies aimed at addressing climate change.\n"
     ]
    }
   ],
   "source": [
    "print(subtopic_responses[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use our `convert_response_to_yaml_list` to clean this up!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "while True:\n",
    "    try:\n",
    "        subtopic_list = generator.convert_response_to_yaml_list(\n",
    "            subtopic_responses[0], model=model, model_kwargs=model_kwargs\n",
    "        )\n",
    "        break\n",
    "    except YamlConversionError as e:\n",
    "        print(f\"Hit: {e}, Retrying...\")\n",
    "        subtopic_responses = generator.generate_subtopics(\n",
    "            macro_topic=topic_list[0], n_subtopics=n_subtopics, model=model\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Global Warming and the Role of Greenhouse Gases',\n",
       " 'Impact of Deforestation on Biodiversity and Climate Change',\n",
       " 'Climate Change and Ocean Acidification',\n",
       " 'Renewable Energy Sources and Sustainable Future',\n",
       " 'Climate Change Mitigation and Adaptation Strategies']"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subtopic_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Generating a Math problem\n",
    "\n",
    "We can now generate Math problems based on the generated topics/subtopics. \n",
    "\n",
    "We can look at the default prompt to see how these questions are generated as we did with the other stages:\n",
    "\n",
    "```python\n",
    "'Generate {n_openlines} mathematics problems which are related to \"{topic}\" or can be addressed using \"{topic}\". Your answer should be a list of problems. Make them as diverse as possible.'\n",
    "```\n",
    "\n",
    "Generating the Math problems is as easy as utilizing the [`generate_math_problem`](https://github.com/NVIDIA/NeMo-Curator/blob/cd4c4907bd4d87cd11d0f37be4ae0fe167a79696/nemo_curator/synthetic/nemotron.py#L471) method.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "question_responses = generator.generate_math_problem(\n",
    "    topic=subtopic_list[0],\n",
    "    n_openlines=10,\n",
    "    model=model\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, we will need to clean our list of questions using the `convert_response_to_yaml_list` method of our generator!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hit: Conversion introduced hallucinations. Original response:\n",
      "1. If the current rate of carbon dioxide emissions is 50 billion tons per year and the concentration of CO2 in the atmosphere is currently 400 parts per million (ppm), assuming no removal or absorption, how many years will it take for the CO2 concentration to reach 500 ppm?\n",
      "2. The Earth absorbs 24% of the solar energy it receives, while the rest is reflected back into space. If greenhouse gases cause the Earth to retain an additional 0.3% of the solar energy, what is the total percentage of solar energy that the Earth now retains?\n",
      "3. If a factory releases 10,000 tons of CO2 per year and can be converted to use renewable energy, which would reduce its emissions to zero, how much will the global CO2 concentration decrease if the factory's emissions are completely eliminated after 10 years?\n",
      "4. The greenhouse effect is responsible for trapping 0.03% of the total solar energy that reaches the Earth's surface. If the concentration of greenhouse gases in the atmosphere increases by 50%, how much more solar energy will be trapped, assuming a linear relationship?\n",
      "5. Assume that the current global temperature increase due to greenhouse gas emissions is 0.01°C per year. If the total greenhouse gas emissions were to be reduced by 25% in the next 10 years, by how much would the temperature increase be reduced over the following 10 years?\n",
      "6. Given that the average American produces 16.5 metric tons of CO2 per year, what percentage reduction in CO2 emissions would be needed to achieve the goal of keeping the global temperature increase below 1.5°C above pre-industrial levels, assuming all other factors remain constant?\n",
      "7. If the global methane concentration is currently 1.8 parts per billion (ppb) and increases by 0.02 ppb per year due to human activities, how many years will it take for the methane concentration to reach 2.0 ppb, assuming no removal or absorption?\n",
      "8. Assume that the current rate of deforestation releases 2.4 billion tons of CO2 per year. If all deforestation were stopped immediately, how much would the global CO2 concentration decrease after 50 years, assuming no other changes in emissions?\n",
      "9. If a new technology can capture and store 90% of the CO2 emissions from a power plant, and the power plant emits 10,000 tons of CO2 per year, how much CO2 would be released into the atmosphere each year with the new technology?\n",
      "10. Given that the global average temperature has increased by approximately 1°C since the pre-industrial era, and that this temperature increase is due to a 40% increase in the concentration of greenhouse gases, estimate the global average temperature increase if the concentration of greenhouse gases were to double.\n",
      "Converted response:\n",
      "['50 billion tons per year', '24.3%', '10,000 tons per year', '0.015%', '0.005°C per year', '66.25%', '55.56 years', '1.2 billion tons of CO2', '1,000 tons of CO2 per year', '2°C']\n",
      "Hallucination:\n",
      "24.3%, Retrying with fewer examples...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Carbon Footprint Calculation',\n",
       " 'Greenhouse Gas Concentration Trends',\n",
       " 'Global Temperature Change Estimation',\n",
       " 'Absorption of Solar Radiation',\n",
       " \"Climate Modeling a City's Temperature Increase\"]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "while True:\n",
    "    try:\n",
    "        question_list = generator.convert_response_to_yaml_list(\n",
    "            question_responses[0], model=model, model_kwargs=model_kwargs\n",
    "        )\n",
    "        break\n",
    "    except YamlConversionError as e:\n",
    "        print(f\"Hit: {e}, Retrying with fewer examples...\")\n",
    "        question_responses = generator.generate_math_problem(\n",
    "            topic=subtopic_list[0],\n",
    "            n_openlines=5,\n",
    "            model=model\n",
    "        )\n",
    "question_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Modifying the Prompts\n",
    "\n",
    "Nemo Curator gives us granular control of each of the prompts at every step of each pipeline - let's look at how we can modify the prompts!\n",
    "\n",
    "We'll start with a simple example of modifying the prompt to another provided default.\n",
    "\n",
    "You can find all available pre-constructed prompts [here](https://github.com/NVIDIA/NeMo-Curator/blob/main/nemo_curator/synthetic/prompts.py).\n",
    "\n",
    "> NOTE: It's important that when you're constructing new prompts you need to include the same placeholders (`{topic}`, `{n_openlines}`, etc.) to ensure smooth integration with `NemotronGenerator`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Using Alternative Prompts\n",
    "\n",
    "Let's examine the `MATH_PROBLEM_BEGINNER_PROMPT_TEMPLATE`:\n",
    "\n",
    "```python\n",
    "'Generate {n_openlines} mathematics problems which are related to \"{topic}\" or can be addressed using \"{topic}\". These problems should be suitable for beginners who just learnt \"{topic}\". Your answer should be a list of problems. Make them as diverse as possible.'\n",
    "```\n",
    "\n",
    "Replacing our existing prompt template with this new one is as easy as including it in our `prompt_template` parameter in the `generate_math_problem` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nemo_curator.synthetic import MATH_PROBLEM_BEGINNER_PROMPT_TEMPLATE  \n",
    "\n",
    "easy_question_responses = generator.generate_math_problem(\n",
    "    topic=subtopic_list[1],\n",
    "    n_openlines=10,\n",
    "    model=model,\n",
    "    prompt_template=MATH_PROBLEM_BEGINNER_PROMPT_TEMPLATE\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, notice that our response needs to be cleaned - and while we can use the `convert_response_to_yaml_list` method to help us, we can also produce custom parsing functions if required. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"1. If a forest covering 10,000 square kilometers is cut down, approximately how many trees are lost? (Assuming an average of 500 trees per hectare and 1 hectare = 0.01 square kilometers)\\n2. If deforestation continues at the current rate, how many years will it take for the world's rainforests to disappear completely? (Assuming the current rate is 150,000 square kilometers per year and the total area of rainforests is 11,500,000 square kilometers)\\n3. If the average temperature increases by 0.2°C for every 1% decrease in forest cover, what will be the increase in temperature if 2% of the forest cover is lost?\\n4. If a country has a carbon footprint of 500 million tons per year and decides to reduce it by planting trees that absorb 10,000 tons of carbon dioxide per square kilometer per year, how many square kilometers of forest would need to be planted to offset the entire carbon footprint?\\n5. If a forest provides habitat for 400 species of birds and 30% of those species are threatened by deforestation, how many bird species are at risk?\\n6. If a logging company harvests trees from a 500-hectare forest every 20 years, what is the annual deforestation rate?\\n7. If 100,000 tons of carbon dioxide are released into the atmosphere each day due to deforestation, how many tons of carbon dioxide are released in a year (assuming 365 days in a year)?\\n8. If 20% of the Amazon rainforest has been destroyed and the Amazon holds 400 billion tons of carbon, how much carbon has been released into the atmosphere due to deforestation?\\n9. If a forest serves as a watershed for a city of 1 million people, what is the impact on the city's water supply if 50% of the forest is lost?\\n10. If planting trees can increase biodiversity, and a tree species has 500 seeds per kilogram and each seed grows into a new tree, how many new trees can be created from 100 kilograms of seeds?\""
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "easy_question_responses[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll define a simple parsing function that will take our `str` response, split it into lines, and then remove the `1.`, `2,`, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['If a forest covering 10,000 square kilometers is cut down, approximately how many trees are lost? (Assuming an average of 500 trees per hectare and 1 hectare = 0.01 square kilometers)',\n",
       " \"If deforestation continues at the current rate, how many years will it take for the world's rainforests to disappear completely? (Assuming the current rate is 150,000 square kilometers per year and the total area of rainforests is 11,500,000 square kilometers)\",\n",
       " 'If the average temperature increases by 0.2°C for every 1% decrease in forest cover, what will be the increase in temperature if 2% of the forest cover is lost?',\n",
       " 'If a country has a carbon footprint of 500 million tons per year and decides to reduce it by planting trees that absorb 10,000 tons of carbon dioxide per square kilometer per year, how many square kilometers of forest would need to be planted to offset the entire carbon footprint?',\n",
       " 'If a forest provides habitat for 400 species of birds and 30% of those species are threatened by deforestation, how many bird species are at risk?',\n",
       " 'If a logging company harvests trees from a 500-hectare forest every 20 years, what is the annual deforestation rate?',\n",
       " 'If 100,000 tons of carbon dioxide are released into the atmosphere each day due to deforestation, how many tons of carbon dioxide are released in a year (assuming 365 days in a year)?',\n",
       " 'If 20% of the Amazon rainforest has been destroyed and the Amazon holds 400 billion tons of carbon, how much carbon has been released into the atmosphere due to deforestation?',\n",
       " \"If a forest serves as a watershed for a city of 1 million people, what is the impact on the city's water supply if 50% of the forest is lost?\",\n",
       " 'If planting trees can increase biodiversity, and a tree species has 500 seeds per kilogram and each seed grows into a new tree, how many new trees can be created from 100 kilograms of seeds?']"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "def parse_math_problem_response(response):\n",
    "    response = response.split(\"\\n\")\n",
    "    return [re.sub(r\"^\\d+\\.\\s\", \"\", line) for line in response]\n",
    "\n",
    "easy_question_list = parse_math_problem_response(easy_question_responses[0])\n",
    "easy_question_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Creating Custom Prompts\n",
    "\n",
    "We can also define our own custom prompts - again, making sure that the placeholder variables are consistent between the original prompt and the newly constructed prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "DIFFICULT_MATH_PROMPT = 'Generate {n_openlines} mathematics problems which are related to \"{topic}\" or can be addressed using \"{topic}\". These problems should be extremely advanced and only solvable by experts who have spent many years learning \"{topic}\". Your answer should be a list of problems, do not name the problems. Make them as diverse as possible.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, we'll generate and then parse into a list!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Develop a mathematical model to quantify the relationship between deforestation, carbon sequestration, and the global carbon budget, taking into account the impacts on biodiversity and climate change',\n",
       " 'Analyze the impact of various deforestation scenarios on species diversity and extinction rates using advanced mathematical techniques such as population dynamics models and biodiversity indices',\n",
       " 'Create a complex mathematical function to estimate the change in surface temperature and precipitation patterns as a result of deforestation-induced climate change',\n",
       " 'Utilize statistical methods to assess the correlation between deforestation rates and changes in local and regional climate patterns, controlling for other factors such as land use and anthropogenic emissions',\n",
       " 'Develop a mathematical framework to evaluate the optimal balance between deforestation for land use and the preservation of biodiversity and climate stability',\n",
       " 'Use mathematical modeling to predict the long-term impacts of deforestation on the global carbon cycle and the feedback loops between carbon sinks and sources',\n",
       " 'Utilize probability theory and stochastic processes to model the variability and uncertainty in the relationship between deforestation, biodiversity, and climate change',\n",
       " 'Develop a mathematical method for integrating the impacts of deforestation on biodiversity and climate change into economic cost-benefit analyses and decision-making frameworks',\n",
       " 'Use optimization algorithms to identify the most effective strategies for reducing deforestation and mitigating its impacts on biodiversity and climate change',\n",
       " 'Use differential equations to model the complex interactions between deforestation, biodiversity loss, and climate change, and develop strategies for managing these systems in a sustainable manner']"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# generate difficult math problems\n",
    "difficult_question_responses = generator.generate_math_problem(\n",
    "    topic=subtopic_list[1],\n",
    "    n_openlines=10,\n",
    "    model=model,\n",
    "    prompt_template=DIFFICULT_MATH_PROMPT\n",
    ")\n",
    "while True:\n",
    "    try:\n",
    "        difficult_question_list = generator.convert_response_to_yaml_list(\n",
    "            difficult_question_responses[0], model=model, model_kwargs=model_kwargs\n",
    "        )\n",
    "        break\n",
    "    except YamlConversionError as e:\n",
    "        print(f\"Hit: {e}, Retrying with fewer examples...\")\n",
    "        difficult_question_responses = generator.generate_math_problem(\n",
    "            topic=subtopic_list[1],\n",
    "            n_openlines=5,\n",
    "            model=model,\n",
    "            prompt_template=DIFFICULT_MATH_PROMPT\n",
    "        )\n",
    "difficult_question_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Async OpenAI Client Usage\n",
    "\n",
    "Now that we've explored a single deconstructed pipeline, we'll work through a number of fantastic built-in pipelines that can be used for a variety of tasks. \n",
    "\n",
    "Before doing that, however, we'll instantiate an asyncronous client and generatore to allow us to generate responses more efficiently!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import AsyncOpenAI\n",
    "from nemo_curator import AsyncOpenAIClient\n",
    "from nemo_curator.synthetic import AsyncNemotronGenerator\n",
    "\n",
    "openai_client = AsyncOpenAI(\n",
    "    base_url=\"https://integrate.api.nvidia.com/v1\", api_key=os.environ[\"NVIDIA_API_KEY\"]\n",
    ")\n",
    "client = AsyncOpenAIClient(openai_client)\n",
    "generator = AsyncNemotronGenerator(client, max_concurrent_requests=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll be leveraging the Nemotron-4 340B Instruct model for the built-in pipelines - but you can substitute any model that is compatible with the OpenAI API spec."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = \"nvidia/nemotron-4-340b-instruct\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Built-In SDG Pipelines\n",
    "\n",
    "- [Math Question Generation Pipeline](#math-question-generation-pipeline)\n",
    "- [Writing Task Generation Pipeline](#writing-task-generation-pipeline)\n",
    "- [Open Question Generation Pipeline](#open-question-pipeline)\n",
    "- [Closed Question Generation Pipeline](#closed-question-pipeline)\n",
    "- [Python Question Generation Pipeline](#python-question-generation-pipeline)\n",
    "- [Dialogue Generation Pipeline](#dialogue-generation-pipeline)\n",
    "- [Two-Turn Prompt Generation Pipeline](#two-turn-prompt-generation-pipeline)\n",
    "- [Entity Classification](#entity-classification)\n",
    "    - [Classify Math Entity](#classify-math-entity)\n",
    "    - [Classify Python Entity](#classify-python-entity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Note on `ignore_conversion_failure=True`\n",
    "\n",
    "Due to the variety of models, and the variety of prompts - conversion from a `str` output to a Python `list` will not always be successful. Due to this, it is currently suggested to set `ignore_conversion_failure=True` to avoid the pipeline breaking down during generation. \n",
    "\n",
    "This will impact the total number of generated entities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Math Question Generation Pipeline\n",
    "\n",
    "The [`run_math_pipeline`](https://github.com/NVIDIA/NeMo-Curator/blob/cd4c4907bd4d87cd11d0f37be4ae0fe167a79696/nemo_curator/synthetic/nemotron.py#L1151) can be used to generate Math questions at various school levels. \n",
    "\n",
    "> NOTE: The `school_level` parameter will influence the generation of Macro Topics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Let (X,Σ,μ) be a measure space and let f:X→ℝ be a Σ-measurable function. Prove that the set {x∈X:f(x)≥t} is Σ-measurable for all t∈ℝ.\n"
     ]
    }
   ],
   "source": [
    "math_questions = await generator.run_math_pipeline(\n",
    "    n_macro_topics=5,\n",
    "    school_level=\"university\",\n",
    "    n_subtopics=5,\n",
    "    n_openlines=10,\n",
    "    model=model,\n",
    "    ignore_conversion_failure=True\n",
    ")\n",
    "print(math_questions[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Writing Task Generation Pipeline\n",
    "\n",
    "The [`run_writing_pipeline`](https://github.com/NVIDIA/NeMo-Curator/blob/cd4c4907bd4d87cd11d0f37be4ae0fe167a79696/nemo_curator/synthetic/nemotron.py#L975) can be used to generate various forms of writing tasks based on provided topics. \n",
    "\n",
    "> NOTE: You could use a topic generation pipeline to generate the seed topics for this pipeline. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:30<00:00,  7.71s/it]\n",
      "100%|██████████| 1/1 [00:30<00:00, 30.83s/it]\n",
      "100%|██████████| 10/10 [01:03<00:00,  6.30s/it]\n",
      "100%|██████████| 10/10 [00:56<00:00,  5.63s/it]\n",
      "100%|██████████| 2/2 [01:59<00:00, 59.68s/it]\n"
     ]
    }
   ],
   "source": [
    "writing_tasks = await generator.run_writing_pipeline(\n",
    "    topics=[\n",
    "        \"Climate Change and Sustainable Living\",\n",
    "        \"Space Exploration and the Universe\",\n",
    "    ],\n",
    "    text_material_types=[\"Poems\", \"Essays\"],\n",
    "    n_openlines=5,\n",
    "    n_revisions=2,\n",
    "    model=model,\n",
    "    ignore_conversion_failure=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Compose a 14-line sonnet in iambic pentameter, praising the beauty and importance of wind and solar power in sustainable living. The sonnet must include at least two concrete examples of how these renewable energy sources contribute to reducing carbon emissions and a reference to their growing global capacity. (Requirement 1, 2, 3)',\n",
       " 'Create a sonnet that highlights the role of wind and solar power in combating climate change, using vivid imagery to describe their functionality and aesthetics. The poem should incorporate at least one data point about the increasing affordability of these technologies and mention a specific region or country that has made significant strides in renewable energy adoption. The sonnet must adhere to the traditional rhyme scheme and contain no more than 150 words. (Requirement 1, 2, 3, 4)',\n",
       " 'Write a 500-word essay discussing the impact of climate change on global food security, focusing on how rising temperatures and shifting precipitation patterns affect crop yields and agricultural productivity. Include data from recent studies and provide examples of regions most vulnerable to these changes. Additionally, propose three sustainable agricultural practices that can help mitigate this issue, such as agroforestry, conservation agriculture, and precision farming, and explain their benefits.',\n",
       " 'In a 3-page, APA-style essay, analyze the relationship between climate change and food insecurity, emphasizing the challenges faced by smallholder farmers in developing countries. Use at least three scholarly sources to support your discussion. Furthermore, recommend two policy measures and one technological solution to promote climate-resilient agriculture, ensuring to address potential barriers to implementation and adoption in your analysis.',\n",
       " 'Write a 300-word essay comparing the efficiency and environmental impact of solar panels and wind turbines in reducing carbon emissions, including real-world examples of their implementation and data on their energy production capabilities. Additionally, discuss the challenges and benefits of integrating these renewable energy sources into existing power grids.']"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "writing_tasks[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Open Question Pipeline\n",
    "\n",
    "The [`run_open_qa_pipeline`](https://github.com/NVIDIA/NeMo-Curator/blob/cd4c4907bd4d87cd11d0f37be4ae0fe167a79696/nemo_curator/synthetic/nemotron.py#L799) can be used to create open questions about desired topics and subtopics. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prompt Modification at Pipeline Level\n",
    "\n",
    "You can freely adjust the prompts, even at the pipeline level!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 10/10 [00:52<00:00,  5.27s/it]\n",
      "100%|██████████| 1/1 [00:52<00:00, 52.70s/it]\n",
      "100%|██████████| 10/10 [01:25<00:00,  8.56s/it]\n",
      "100%|██████████| 10/10 [01:33<00:00,  9.39s/it]\n",
      "100%|██████████| 4/4 [00:58<00:00, 14.55s/it]\n",
      "100%|██████████| 3/3 [03:57<00:00, 79.24s/it]\n",
      "100%|██████████| 10/10 [01:24<00:00,  8.42s/it]\n",
      "100%|██████████| 10/10 [01:13<00:00,  7.40s/it]\n",
      "100%|██████████| 10/10 [00:56<00:00,  5.68s/it]\n",
      "100%|██████████| 10/10 [01:01<00:00,  6.11s/it]\n",
      "100%|██████████| 10/10 [00:48<00:00,  4.89s/it]\n",
      "100%|██████████| 10/10 [01:12<00:00,  7.22s/it]\n",
      "100%|██████████| 10/10 [00:51<00:00,  5.19s/it]\n",
      "100%|██████████| 10/10 [00:48<00:00,  4.83s/it]\n",
      "100%|██████████| 10/10 [01:04<00:00,  6.50s/it]\n",
      "100%|██████████| 10/10 [00:53<00:00,  5.35s/it]\n",
      "100%|██████████| 10/10 [00:53<00:00,  5.35s/it]\n",
      "100%|██████████| 10/10 [00:49<00:00,  4.94s/it]\n",
      "100%|██████████| 12/12 [11:58<00:00, 59.89s/it]\n"
     ]
    }
   ],
   "source": [
    "# define new open QA prompt\n",
    "NEW_OPEN_QA_PROMPT = \"\"\"\\\n",
    "Can you generate {n_openlines} questions or requests related to {topic}? The questions should build off eachother. Your answer should be a list.\n",
    "\"\"\"\n",
    "\n",
    "# run open QA pipeline\n",
    "open_qa_questions = await generator.run_open_qa_pipeline(\n",
    "    n_macro_topics=1,\n",
    "    n_subtopics=2,\n",
    "    n_openlines=5,\n",
    "    n_revisions=2,\n",
    "    model=model,\n",
    "    open_qa_from_topics_prompt_template=NEW_OPEN_QA_PROMPT, # substitute the default prompt with the new prompt\n",
    "    ignore_conversion_failure=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Given the urgent need to reduce carbon emissions and promote sustainable urban development, could you provide examples of cities or regions that have successfully implemented green transportation technologies, such as electric buses or bike-sharing systems? In your response, please detail the specific infrastructure changes these cities made to support these innovative solutions, like installing charging stations or creating dedicated bike lanes.'"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "open_qa_questions[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Closed Question Pipeline\n",
    "\n",
    "You can use the [`run_closed_qa_pipeline`](https://github.com/NVIDIA/NeMo-Curator/blob/cd4c4907bd4d87cd11d0f37be4ae0fe167a79696/nemo_curator/synthetic/nemotron.py#L1082) to generate questions specific to a provided context. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll be using a snippet of an NVIDIA Blog as our context for this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [],
   "source": [
    "blog_text = \"\"\"\\\n",
    "NVIDIA today announced Nemotron-4 340B, a family of open models that developers can use to generate synthetic data for training large language models (LLMs) for commercial applications across healthcare, finance, manufacturing, retail and every other industry.\n",
    "\n",
    "High-quality training data plays a critical role in the performance, accuracy and quality of responses from a custom LLM — but robust datasets can be prohibitively expensive and difficult to access.\n",
    "\n",
    "Through a uniquely permissive open model license, Nemotron-4 340B gives developers a free, scalable way to generate synthetic data that can help build powerful LLMs.\n",
    "\n",
    "The Nemotron-4 340B family includes base, instruct and reward models that form a pipeline to generate synthetic data used for training and refining LLMs. The models are optimized to work with NVIDIA NeMo, an open-source framework for end-to-end model training, including data curation, customization and evaluation. They’re also optimized for inference with the open-source NVIDIA TensorRT-LLM library.\n",
    "\n",
    "Nemotron-4 340B can be downloaded now from the NVIDIA NGC catalog and from Hugging Face, where developers can also use the Train on DGX Cloud service to easily fine-tune open AI models. Developers will soon be able to access the models at ai.nvidia.com, where they’ll be packaged as an NVIDIA NIM microservice with a standard application programming interface that can be deployed anywhere.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of the pipeline is in tuple format:\n",
    "\n",
    "```python\n",
    "[\n",
    "    (0, \"Sample Question About Document at Index 0\"),\n",
    "    ...,\n",
    "    (1, \"Sample Question ABout Document at Index 1\"),\n",
    "    ...,\n",
    "    (2, \"Sample Question ABout Document at Index 2\")\n",
    "]\n",
    "```\n",
    "\n",
    "Where the 1st element of the tuple refers to the index of the document the question (in the 2nd element of the tuple) pertains to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [01:14<00:00, 74.29s/it]\n",
      "100%|██████████| 1/1 [01:14<00:00, 74.29s/it]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  \"Can you summarize the main purpose of NVIDIA's newly announced Nemotron-4 340B in one sentence?\"),\n",
       " (0,\n",
       "  'Explain how the Nemotron-4 340B family of models can help developers create custom LLMs for various industries.'),\n",
       " (0,\n",
       "  'Write a short paragraph about the significance of high-quality training data in the development of LLMs and how Nemotron-4 340B addresses this challenge.'),\n",
       " (0,\n",
       "  'How does NVIDIA NeMo and TensorRT-LLM library relate to Nemotron-4 340B, and what benefits do they provide to developers?'),\n",
       " (0,\n",
       "  'Create a tweet announcing the release of Nemotron-4 340B, highlighting its key features and benefits for developers.'),\n",
       " (0,\n",
       "  'Identify the three types of models included in the Nemotron-4 340B family and explain their roles in the synthetic data generation pipeline.'),\n",
       " (0,\n",
       "  'Rephrase the section about the availability of Nemotron-4 340B, focusing on the various platforms where developers can access and utilize the models.'),\n",
       " (0,\n",
       "  'Compare and contrast the process of training LLMs with and without the use of synthetic data generated by Nemotron-4 340B.'),\n",
       " (0,\n",
       "  'Write a brief blog post introduction discussing the importance of open models like Nemotron-4 340B in democratizing AI and fostering innovation.'),\n",
       " (0,\n",
       "  'Design a simple infographic that illustrates the workflow of using Nemotron-4 340B to generate synthetic data and train custom LLMs.')]"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "closed_qa_questions = await generator.run_closed_qa_pipeline(\n",
    "    documents=[blog_text], # pass the blog text as a list\n",
    "    n_openlines=10,\n",
    "    model=model,\n",
    "    ignore_conversion_failure=True\n",
    ")\n",
    "closed_qa_questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python Question Generation Pipeline\n",
    "\n",
    "The [`run_python_pipeline`](https://github.com/NVIDIA/NeMo-Curator/blob/cd4c4907bd4d87cd11d0f37be4ae0fe167a79696/nemo_curator/synthetic/nemotron.py#L1296) can be used to generate questions pertaining to Python tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [01:26<00:00, 28.78s/it]\n",
      "100%|██████████| 1/1 [01:26<00:00, 86.35s/it]\n",
      "100%|██████████| 3/3 [02:02<00:00, 40.88s/it]\n",
      "100%|██████████| 1/1 [02:02<00:00, 122.65s/it]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Write a Python program to create a stack data structure using a list and implement the push, pop, and is_empty methods.',\n",
       " 'Create a program that uses a queue data structure (implemented with a list) to manage a simple printing queue. The program should allow users to add print tasks to the queue, process the tasks one by one, and display the current queue status.',\n",
       " 'Implement a Python program that uses a dictionary to count the frequency of each word in a given string. The program should also print the words in descending order of their frequency.',\n",
       " 'Write a Python program to create a linked list data structure and implement the append, insert, and delete methods. Additionally, create a method to display the linked list elements.',\n",
       " 'Develop a Python program that uses a set data structure to find the union, intersection, and difference between two given lists of numbers.']"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "python_questions = await generator.run_python_pipeline(\n",
    "    n_macro_topics=3,\n",
    "    n_subtopics=2,\n",
    "    n_openlines=10,\n",
    "    model=model,\n",
    "    ignore_conversion_failure=True\n",
    ")\n",
    "python_questions[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dialogue Generation Pipeline\n",
    "\n",
    "This pipeline will take existing openlines (generated by any of the above pipelines, as an example) and convert them into dialogues or multiple turn conversations:\n",
    "\n",
    "- [`generate_dialogue`](https://github.com/NVIDIA/NeMo-Curator/blob/cd4c4907bd4d87cd11d0f37be4ae0fe167a79696/nemo_curator/synthetic/nemotron.py#L654): This will generate `n` number of turns between a User and an Assistant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "dialogue = await generator.generate_dialogue(\n",
    "    openline=\"Write a poem about the moon.\",\n",
    "    user_model=model,\n",
    "    assistant_model=model,\n",
    "    n_user_turns=3,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'role': 'user', 'content': 'Write a poem about the moon.'},\n",
       " {'role': 'assistant',\n",
       "  'content': \"In the velvet night, a silver orb ascends,\\nA quiet guardian, on whom the stars depend.\\nThe moon, with her gentle, radiant grace,\\nGuides us through the darkness, with her soft, lunar face.\\n\\nShe's a cosmic poet, whispering tales of old,\\nOf ancient seas and craters, stories to be told.\\nHer phases mark the passage of time, a celestial clock,\\nFrom crescent to full, in her grand, cyclical walk.\\n\\nShe inspires the tides, with her gravitational pull,\\nA dance between the earth and moon, beautiful and full.\\nHer light, though borrowed, is never less,\\nA beacon of hope, in the abyss of endless space.\\n\\nTo lovers, she's a confidant, bearing silent witness,\\nTo promises made under her soft, ethereal kiss.\\nTo dreamers, she's a muse, igniting sparks of creativity,\\nA canvas for imagination, boundless and free.\\n\\nSo here's to the moon, our companion in the night,\\nA symbol of mystery, bathed in soft, silver light.\\nMay she continue to inspire, to guide, and to soothe,\\nA timeless friend, in the grand theater of the universe's roof.\\n\"},\n",
       " {'role': 'user',\n",
       "  'content': 'Could you now compose a poem about the sun, with a similar tone and length, showcasing its power, importance, and the life it brings?\\n'},\n",
       " {'role': 'assistant',\n",
       "  'content': \"In the grand theater of the sky, a golden orb ascends,\\nThe sun, with its fiery, life-giving power, on which all life depends.\\nA blazing sovereign, ruling the azure dome,\\nBringing warmth and light, to every earthly home.\\n\\nIt's a cosmic artist, painting sunsets and dawns,\\nWith hues of crimson, orange, and gold, on horizons it adorns.\\nIts rays, like fingers, reach through the clouds,\\nAwakening the world, dispelling night's shrouds.\\n\\nThe sun is a giver, of energy and life,\\nTo plants and creatures, in their daily strife.\\nIts light, a catalyst, for growth and photosynthesis,\\nA symphony of life, in every verdant abyss.\\n\\nTo farmers, it's a blessing, ripening fields of grain,\\nTo sailors, a compass, guiding them through the main.\\nTo children, it's a playmate, in games of shadow and light,\\nA source of joy, in the warmth of its inviting sight.\\n\\nSo here's to the sun, our radiant, celestial king,\\nA symbol of power, life, and the joy that it brings.\\nMay it continue to radiate, to nourish, and to inspire,\\nA timeless beacon, in the grand cosmic choir.\\n\\nYet, let us not forget, to respect its might,\\nFor its fiery embrace, can be as harsh as it is bright.\\nA delicate balance, between life and death,\\nThe sun, a reminder, of our shared, planetary breath.\\n\"},\n",
       " {'role': 'user',\n",
       "  'content': 'Could you now create a poem about the earth, connecting it with the moon and sun, emphasizing their interdependence and the delicate balance they maintain for life to flourish?\\n'},\n",
       " {'role': 'assistant',\n",
       "  'content': \"In the cosmic dance, three celestial bodies entwine,\\nThe Earth, the Moon, and Sun, in a rhythmic, harmonious line.\\nA trinity of existence, each playing a vital role,\\nIn the grand, intricate symphony, of life's eternal goal.\\n\\nThe Earth, a nurturing mother, teeming with life,\\nA vibrant, blue oasis, amidst the cosmic strife.\\nShe's a canvas for the sun's warm, golden touch,\\nAnd the moon's soft, silver kiss, in the hush of night's clutch.\\n\\nThe sun, a fiery father, bestowing energy and light,\\nGuiding the seasons, in their cyclical flight.\\nHis rays, a lifeline, for the Earth's myriad forms,\\nA dance of photosynthesis, in forests and farms.\\n\\nThe moon, a gentle sister, regulating tides and time,\\nA silent guardian, in the grand, cosmic rhyme.\\nHer phases, a metronome, for life's rhythmic beat,\\nA soothing balm, in the night's serene retreat.\\n\\nTogether, they maintain a delicate balance, a celestial pact,\\nEach giving and taking, in a mutual, harmonious act.\\nThe Earth, the Sun, and Moon, in a waltz of interdependence,\\nA testament to the beauty, of nature's grand alliance.\\n\\nYet, this balance is fragile, requiring our utmost care,\\nFor the health of our Earth, is beyond compare.\\nLet us honor this trinity, with respect and love,\\nA sacred trust, in the vastness above.\\n\\nSo here's to the Earth, the Sun, and the Moon,\\nA celestial family, in the grand, cosmic swoon.\\nMay we cherish their bond, and the life it sustains,\\nIn the grand, interconnected dance, of the universe's veins.\\n\"}]"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dialogue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Two-Turn Prompt Generation Pipeline\n",
    "\n",
    "This pipeline will take, as input, a single openline - and it will return an Assistant response, as well as a synthetically generated User response. This can be used as a prompt for a two-turn Assistant response.\n",
    "\n",
    "- [`generate_two_turn_prompt`](https://github.com/NVIDIA/NeMo-Curator/blob/cd4c4907bd4d87cd11d0f37be4ae0fe167a79696/nemo_curator/synthetic/nemotron.py#L721): This will generate: \"User (Input) -> Assistant (Output) -> User (Output)\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "dialogue = await generator.generate_two_turn_prompt(\n",
    "    openline=\"Write a poem about the moon.\",\n",
    "    user_model=model,\n",
    "    assistant_model=model,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'role': 'user', 'content': 'Write a poem about the moon.'},\n",
       " {'role': 'assistant',\n",
       "  'content': \"In the velvet night, a silver orb ascends,\\nA quiet guardian, on whom the darkness depends.\\nThe moon, with her gentle, ethereal glow,\\nGuides us through shadows, in her soothing flow.\\n\\nCraters and valleys, a face etched in time,\\nA silent witness to the cosmos' rhyme.\\nHer phases a dance, from crescent to full,\\nIn her rhythmic cycle, she pulls and she lull.\\n\\nShe whispers to lovers, under her soft light,\\nInspiring poetry in the stillness of night.\\nTo the tides she sings, a celestial song,\\nIn her gravitational embrace, they belong.\\n\\nA beacon for sailors, lost in the sea,\\nA companion to dreamers, in their reverie.\\nHer light, though borrowed, is never less,\\nA reflection of the sun, in her elegant dress.\\n\\nSo here's to the moon, our celestial guide,\\nIn her we find peace, in her we confide.\\nThrough the ebb and flow of life's vast sea,\\nShe's a constant reminder, of the beauty that's free.\\n\"},\n",
       " {'role': 'user',\n",
       "  'content': 'Could you now compose a poem about the sun, as a companion piece to the moon poem?\\n'}]"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dialogue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Entity Classification\n",
    "\n",
    "Aside from generating synthetic responses - the NeMo Curator SDG toolset can also be used to verify or identify if particular entities are related to Math, or Python out of the box. \n",
    "\n",
    ">NOTE: This process can be extended to a more diverse set of topics by modifying the prompt templates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classify Math Entity\n",
    "\n",
    "The prompt used for this task is as follows:\n",
    "\n",
    "```python\n",
    "'Does the concept \"{entity}\" belong to one of the following categories?\\n- Math concepts taught at elementary school, middle school, high school, and univiersity.\\n- Important mathematics axioms, theorems, algorithms, equations, or inequalities.\\n- Representative math problems, functions, and applications.\\n\\nYour answer should start with \"Yes\" or \"No\".'\n",
    "```\n",
    "\n",
    "- [`classify_math_entity`](https://github.com/NVIDIA/NeMo-Curator/blob/cd4c4907bd4d87cd11d0f37be4ae0fe167a79696/nemo_curator/synthetic/nemotron.py#L439): This will classify if an entity is related to math or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Yes, the concept \"What is the formula for the area of a circle?\" belongs to the first category: Math concepts taught at elementary school, middle school, high school, and university. Specifically, the formula for the area of a circle, which is A = πr², is typically taught in middle school or early high school.\\n']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = await generator.classify_math_entity(\n",
    "    entity=\"What is the formula for the area of a circle?\",\n",
    "    model=model\n",
    ")\n",
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['No, the concept \"Pizza Pi is so delicious\" does not belong to any of the listed categories, as it is not a mathematical concept, axiom, theorem, algorithm, equation, inequality, problem, function, or application. It appears to be a subjective statement about the taste of a food item named \"Pizza Pi.\"\\n']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = await generator.classify_math_entity(\n",
    "    entity=\"Pizza Pie is so delicious.\",\n",
    "    model=model\n",
    ")\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classify Python Entity\n",
    "\n",
    "The prompt used for this task is as follows:\n",
    "\n",
    "```python\n",
    "'Does the concept \"{entity}\" belong to one of the following categories?\\n- Programming concepts like loops, functions, and data structures in python.\\n- Important functions, objects, or libraries in python.\\n- Mathematical concepts like linear algebra which can be implemented in python.\\n- Basic algorithms or problems in computer science likes Greedy Search and Dynamics programming which can be addressed in python.\\n\\nYour answer should start with \"Yes\" or \"No\".'\n",
    "```\n",
    "\n",
    "- [`classify_python_entity`](https://github.com/NVIDIA/NeMo-Curator/blob/cd4c4907bd4d87cd11d0f37be4ae0fe167a79696/nemo_curator/synthetic/nemotron.py#L578): This will classify if an entity is related to Python or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Yes, the concept \"How do I write a for loop in Python?\" belongs to the category of \"Programming concepts like loops, functions, and data structures in python.\" For loops are a fundamental control flow statement in Python used for iterating over a sequence (such as a list, tuple, or string) or other iterable objects.\\n']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = await generator.classify_python_entity(\n",
    "    entity=\"How do I write a for loop in Python?\",\n",
    "    model=model\n",
    ")\n",
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['No, the concept \"Pythons are large snakes.\" does not belong to any of the mentioned categories. It is a statement about the biological python species and not related to programming, mathematical concepts, or computer science algorithms in the context of the Python programming language.\\n']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = await generator.classify_python_entity(\n",
    "    entity=\"Pythons are large snakes.\",\n",
    "    model=model\n",
    ")\n",
    "response"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}