Make it easy to run evaluation directly from this repo (#2233)

* Updating docs

* Update requirements.txt

* Update diagram

* Add typing extensions explicitly

* Adding ground truth generation

* Add evaluate flow as well

* Add RAGAS

* Add RAGAS

* Remove simulator

* Improvements to RAGAS code

* More logging, save knowledge graph after transforms

* Update baseline, add citations matched metric, use separate venv for eval

* Update the requirements to latest tag

* Logger fixes

.gitignore

@@ -111,6 +111,7 @@ celerybeat.pid
 # Environments
 .env
 .venv
+.evalenv
 env/
 venv/
 ENV/

docs/evaluation.md

@@ -0,0 +1,104 @@
+# Evaluating the RAG answer quality
+
+Follow these steps to evaluate the quality of the answers generated by the RAG flow.
+
+* [Deploy an evaluation model](#deploy-an-evaluation-model)
+* [Setup the evaluation environment](#setup-the-evaluation-environment)
+* [Generate ground truth data](#generate-ground-truth-data)
+* [Run bulk evaluation](#run-bulk-evaluation)
+* [Review the evaluation results](#review-the-evaluation-results)
+* [Run bulk evaluation on a PR](#run-bulk-evaluation-on-a-pr)
+
+## Deploy an evaluation model
+
+1. Run this command to tell `azd` to deploy a GPT-4 level model for evaluation:
+
+    ```shell
+    azd env set USE_EVAL true
+    ```
+
+2. Set the capacity to the highest possible value to ensure that the evaluation runs relatively quickly. Even with a high capacity, it can take a long time to generate ground truth data and run bulk evaluations.
+
+    ```shell
+    azd env set AZURE_OPENAI_EVAL_DEPLOYMENT_CAPACITY 100
+    ```
+
+    By default, that will provision a `gpt-4o` model, version `2024-08-06`. To change those settings, set the azd environment variables `AZURE_OPENAI_EVAL_MODEL` and `AZURE_OPENAI_EVAL_MODEL_VERSION` to the desired values.
+
+3. Then, run the following command to provision the model:
+
+    ```shell
+    azd provision
+    ```
+
+## Setup the evaluation environment
+
+Make a new Python virtual environment and activate it. This is currently required due to incompatibilities between the dependencies of the evaluation script and the main project.
+
+```bash
+python -m venv .evalenv
+```
+
+```bash
+source .evalenv/bin/activate
+```
+
+Install all the dependencies for the evaluation script by running the following command:
+
+```bash
+pip install -r evals/requirements.txt
+```
+
+## Generate ground truth data
+
+Modify the search terms and tasks in `evals/generate_config.json` to match your domain.
+
+Generate ground truth data by running the following command:
+
+```bash
+python evals/generate_ground_truth.py --numquestions=200 --numsearchdocs=1000
+```
+
+The options are:
+
+* `numquestions`: The number of questions to generate. We suggest at least 200.
+* `numsearchdocs`: The number of documents (chunks) to retrieve from your search index. You can leave off the option to fetch all documents, but that will significantly increase time it takes to generate ground truth data. You may want to at least start with a subset.
+* `kgfile`: An existing RAGAS knowledge base JSON file, which is usually `ground_truth_kg.json`. You may want to specify this if you already created a knowledge base and just want to tweak the question generation steps.
+* `groundtruthfile`: The file to write the generated ground truth answwers. By default, this is `evals/ground_truth.jsonl`.
+
+🕰️ This may take a long time, possibly several hours, depending on the size of the search index.
+
+Review the generated data in `evals/ground_truth.jsonl` after running that script, removing any question/answer pairs that don't seem like realistic user input.
+
+## Run bulk evaluation
+
+Review the configuration in `evals/eval_config.json` to ensure that everything is correctly setup. You may want to adjust the metrics used. See [the ai-rag-chat-evaluator README](https://github.com/Azure-Samples/ai-rag-chat-evaluator) for more information on the available metrics.
+
+By default, the evaluation script will evaluate every question in the ground truth data.
+Run the evaluation script by running the following command:
+
+```bash
+python evals/evaluate.py
+```
+
+🕰️ This may take a long time, possibly several hours, depending on the number of ground truth questions. You can specify `--numquestions` argument for a test run on a subset of the questions.
+
+## Review the evaluation results
+
+The evaluation script will output a summary of the evaluation results, inside the `evals/results` directory.
+
+You can see a summary of results across all evaluation runs by running the following command:
+
+```bash
+python -m evaltools summary evals/results
+```
+
+Compare answers across runs by running the following command:
+
+```bash
+python -m evaltools diff evals/results/baseline/
+```
+
+## Run bulk evaluation on a PR
+
+To run the evaluation on the changes in a PR, you can add a `/evaluate` comment to the PR. This will trigger the evaluation workflow to run the evaluation on the PR changes and will post the results to the PR.

evals/evaluate.py

@@ -0,0 +1,92 @@
+import argparse
+import logging
+import os
+import re
+from pathlib import Path
+
+from azure.identity import AzureDeveloperCliCredential
+from dotenv_azd import load_azd_env
+from evaltools.eval.evaluate import run_evaluate_from_config
+from evaltools.eval.evaluate_metrics import register_metric
+from evaltools.eval.evaluate_metrics.base_metric import BaseMetric
+from rich.logging import RichHandler
+
+logger = logging.getLogger("ragapp")
+
+
+class CitationsMatchedMetric(BaseMetric):
+    METRIC_NAME = "citations_matched"
+
+    @classmethod
+    def evaluator_fn(cls, **kwargs):
+        def citations_matched(*, response, ground_truth, **kwargs):
+            if response is None:
+                logger.warning("Received response of None, can't compute citation_match metric. Setting to -1.")
+                return {cls.METRIC_NAME: -1}
+            # Return true if all citations in the truth are present in the response
+            truth_citations = set(re.findall(r"\[([^\]]+)\.\w{3,4}(#page=\d+)*\]", ground_truth))
+            response_citations = set(re.findall(r"\[([^\]]+)\.\w{3,4}(#page=\d+)*\]", response))
+            # Count the percentage of citations that are present in the response
+            num_citations = len(truth_citations)
+            num_matched_citations = len(truth_citations.intersection(response_citations))
+            return {cls.METRIC_NAME: num_matched_citations / num_citations}
+
+        return citations_matched
+
+    @classmethod
+    def get_aggregate_stats(cls, df):
+        df = df[df[cls.METRIC_NAME] != -1]
+        return {
+            "total": int(df[cls.METRIC_NAME].sum()),
+            "rate": round(df[cls.METRIC_NAME].mean(), 2),
+        }
+
+
+def get_openai_config():
+    azure_endpoint = f"https://{os.getenv('AZURE_OPENAI_SERVICE')}.openai.azure.com"
+    azure_deployment = os.environ["AZURE_OPENAI_EVAL_DEPLOYMENT"]
+    openai_config = {"azure_endpoint": azure_endpoint, "azure_deployment": azure_deployment}
+    # azure-ai-evaluate will call DefaultAzureCredential behind the scenes,
+    # so we must be logged in to Azure CLI with the correct tenant
+    return openai_config
+
+
+def get_azure_credential():
+    AZURE_TENANT_ID = os.getenv("AZURE_TENANT_ID")
+    if AZURE_TENANT_ID:
+        logger.info("Setting up Azure credential using AzureDeveloperCliCredential with tenant_id %s", AZURE_TENANT_ID)
+        azure_credential = AzureDeveloperCliCredential(tenant_id=AZURE_TENANT_ID, process_timeout=60)
+    else:
+        logger.info("Setting up Azure credential using AzureDeveloperCliCredential for home tenant")
+        azure_credential = AzureDeveloperCliCredential(process_timeout=60)
+    return azure_credential
+
+
+if __name__ == "__main__":
+    logging.basicConfig(
+        level=logging.WARNING, format="%(message)s", datefmt="[%X]", handlers=[RichHandler(rich_tracebacks=True)]
+    )
+    logger.setLevel(logging.INFO)
+    logging.getLogger("evaltools").setLevel(logging.INFO)
+    load_azd_env()
+
+    parser = argparse.ArgumentParser(description="Run evaluation with OpenAI configuration.")
+    parser.add_argument("--targeturl", type=str, help="Specify the target URL.")
+    parser.add_argument("--resultsdir", type=Path, help="Specify the results directory.")
+    parser.add_argument("--numquestions", type=int, help="Specify the number of questions.")
+
+    args = parser.parse_args()
+
+    openai_config = get_openai_config()
+
+    register_metric(CitationsMatchedMetric)
+    run_evaluate_from_config(
+        working_dir=Path(__file__).parent,
+        config_path="evaluate_config.json",
+        num_questions=args.numquestions,
+        target_url=args.targeturl,
+        results_dir=args.resultsdir,
+        openai_config=openai_config,
+        model=os.environ["AZURE_OPENAI_EVAL_MODEL"],
+        azure_credential=get_azure_credential(),
+    )

evals/evaluate_config.json

@@ -0,0 +1,28 @@
+{
+    "testdata_path": "ground_truth.jsonl",
+    "results_dir": "results/experiment<TIMESTAMP>",
+    "requested_metrics": ["gpt_groundedness", "gpt_relevance", "answer_length", "latency", "citations_matched"],
+    "target_url": "http://localhost:50505/chat",
+    "target_parameters": {
+        "overrides": {
+            "top": 3,
+            "temperature": 0.3,
+            "minimum_reranker_score": 0,
+            "minimum_search_score": 0,
+            "retrieval_mode": "hybrid",
+            "semantic_ranker": true,
+            "semantic_captions": false,
+            "suggest_followup_questions": false,
+            "use_oid_security_filter": false,
+            "use_groups_security_filter": false,
+            "vector_fields": [
+                "embedding"
+            ],
+            "use_gpt4v": false,
+            "gpt4v_input": "textAndImages",
+            "seed": 1
+        }
+    },
+    "target_response_answer_jmespath": "message.content",
+    "target_response_context_jmespath": "context.data_points.text"
+}

evals/generate_ground_truth.py

@@ -0,0 +1,161 @@
+import argparse
+import json
+import logging
+import os
+import pathlib
+import re
+
+from azure.identity import AzureDeveloperCliCredential, get_bearer_token_provider
+from azure.search.documents import SearchClient
+from dotenv_azd import load_azd_env
+from langchain_core.documents import Document as LCDocument
+from langchain_openai import AzureChatOpenAI, AzureOpenAIEmbeddings
+from ragas.embeddings import LangchainEmbeddingsWrapper
+from ragas.llms import LangchainLLMWrapper
+from ragas.testset import TestsetGenerator
+from ragas.testset.graph import KnowledgeGraph, Node, NodeType
+from ragas.testset.transforms import apply_transforms, default_transforms
+from rich.logging import RichHandler
+
+logger = logging.getLogger("ragapp")
+
+root_dir = pathlib.Path(__file__).parent
+
+
+def get_azure_credential():
+    AZURE_TENANT_ID = os.getenv("AZURE_TENANT_ID")
+    if AZURE_TENANT_ID:
+        logger.info("Setting up Azure credential using AzureDeveloperCliCredential with tenant_id %s", AZURE_TENANT_ID)
+        azure_credential = AzureDeveloperCliCredential(tenant_id=AZURE_TENANT_ID, process_timeout=60)
+    else:
+        logger.info("Setting up Azure credential using AzureDeveloperCliCredential for home tenant")
+        azure_credential = AzureDeveloperCliCredential(process_timeout=60)
+    return azure_credential
+
+
+def get_search_documents(azure_credential, num_search_documents=None) -> str:
+    search_client = SearchClient(
+        endpoint=f"https://{os.getenv('AZURE_SEARCH_SERVICE')}.search.windows.net",
+        index_name=os.getenv("AZURE_SEARCH_INDEX"),
+        credential=azure_credential,
+    )
+    all_documents = []
+    if num_search_documents is None:
+        logger.info("Fetching all document chunks from Azure AI Search")
+        num_search_documents = 100000
+    else:
+        logger.info("Fetching %d document chunks from Azure AI Search", num_search_documents)
+    response = search_client.search(search_text="*", top=num_search_documents).by_page()
+    for page in response:
+        page = list(page)
+        all_documents.extend(page)
+    return all_documents
+
+
+def generate_ground_truth_ragas(num_questions=200, num_search_documents=None, kg_file=None):
+    azure_credential = get_azure_credential()
+    azure_openai_api_version = os.getenv("AZURE_OPENAI_API_VERSION") or "2024-06-01"
+    azure_endpoint = f"https://{os.getenv('AZURE_OPENAI_SERVICE')}.openai.azure.com"
+    azure_ad_token_provider = get_bearer_token_provider(
+        azure_credential, "https://cognitiveservices.azure.com/.default"
+    )
+    generator_llm = LangchainLLMWrapper(
+        AzureChatOpenAI(
+            openai_api_version=azure_openai_api_version,
+            azure_endpoint=azure_endpoint,
+            azure_ad_token_provider=azure_ad_token_provider,
+            azure_deployment=os.getenv("AZURE_OPENAI_EVAL_DEPLOYMENT"),
+            model=os.environ["AZURE_OPENAI_EVAL_MODEL"],
+            validate_base_url=False,
+        )
+    )
+
+    # init the embeddings for answer_relevancy, answer_correctness and answer_similarity
+    generator_embeddings = LangchainEmbeddingsWrapper(
+        AzureOpenAIEmbeddings(
+            openai_api_version=azure_openai_api_version,
+            azure_endpoint=azure_endpoint,
+            azure_ad_token_provider=azure_ad_token_provider,
+            azure_deployment=os.getenv("AZURE_OPENAI_EMB_DEPLOYMENT"),
+            model=os.environ["AZURE_OPENAI_EMB_MODEL_NAME"],
+        )
+    )
+
+    # Load or create the knowledge graph
+    if kg_file:
+        full_path_to_kg = root_dir / kg_file
+        if not os.path.exists(full_path_to_kg):
+            raise FileNotFoundError(f"Knowledge graph file {full_path_to_kg} not found.")
+        logger.info("Loading existing knowledge graph from %s", full_path_to_kg)
+        kg = KnowledgeGraph.load(full_path_to_kg)
+    else:
+        # Make a knowledge_graph from Azure AI Search documents
+        search_docs = get_search_documents(azure_credential, num_search_documents)
+
+        logger.info("Creating a RAGAS knowledge graph based off of %d search documents", len(search_docs))
+        nodes = []
+        for doc in search_docs:
+            content = doc["content"]
+            citation = doc["sourcepage"]
+            node = Node(
+                type=NodeType.DOCUMENT,
+                properties={
+                    "page_content": f"[[{citation}]]: {content}",
+                    "document_metadata": {"citation": citation},
+                },
+            )
+            nodes.append(node)
+
+        kg = KnowledgeGraph(nodes=nodes)
+
+        logger.info("Using RAGAS to apply transforms to knowledge graph")
+        transforms = default_transforms(
+            documents=[LCDocument(page_content=doc["content"]) for doc in search_docs],
+            llm=generator_llm,
+            embedding_model=generator_embeddings,
+        )
+        apply_transforms(kg, transforms)
+
+        kg.save(root_dir / "ground_truth_kg.json")
+
+    logger.info("Using RAGAS knowledge graph to generate %d questions", num_questions)
+    generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings, knowledge_graph=kg)
+    dataset = generator.generate(testset_size=num_questions, with_debugging_logs=True)
+
+    qa_pairs = []
+    for sample in dataset.samples:
+        question = sample.eval_sample.user_input
+        truth = sample.eval_sample.reference
+        # Grab the citation in square brackets from the reference_contexts and add it to the truth
+        citations = []
+        for context in sample.eval_sample.reference_contexts:
+            match = re.search(r"\[\[(.*?)\]\]", context)
+            if match:
+                citation = match.group(1)
+                citations.append(f"[{citation}]")
+        truth += " " + " ".join(citations)
+        qa_pairs.append({"question": question, "truth": truth})
+
+    with open(root_dir / "ground_truth.jsonl", "a") as f:
+        logger.info("Writing %d QA pairs to %s", len(qa_pairs), f.name)
+        for qa_pair in qa_pairs:
+            f.write(json.dumps(qa_pair) + "\n")
+
+
+if __name__ == "__main__":
+    logging.basicConfig(
+        level=logging.WARNING, format="%(message)s", datefmt="[%X]", handlers=[RichHandler(rich_tracebacks=True)]
+    )
+    logger.setLevel(logging.INFO)
+    load_azd_env()
+
+    parser = argparse.ArgumentParser(description="Generate ground truth data using AI Search index and RAGAS.")
+    parser.add_argument("--numsearchdocs", type=int, help="Specify the number of search results to fetch")
+    parser.add_argument("--numquestions", type=int, help="Specify the number of questions to generate.", default=200)
+    parser.add_argument("--kgfile", type=str, help="Specify the path to an existing knowledge graph file")
+
+    args = parser.parse_args()
+
+    generate_ground_truth_ragas(
+        num_search_documents=args.numsearchdocs, num_questions=args.numquestions, kg_file=args.kgfile
+    )