Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support basic text output in tests #1113

Merged
merged 2 commits into from
Feb 8, 2025
Merged

support basic text output in tests #1113

merged 2 commits into from
Feb 8, 2025

Conversation

pelikhan
Copy link
Member

@pelikhan pelikhan commented Feb 8, 2025

Introduce 'format' to PromptTest and refactored transforms logic.


• 📝 Updated test documentation to clarify that GenAIScript now extracts the "text" field by default, with an option to use raw JSON output by specifying format: "json".
• ⚙️ Enhanced test configuration logic to support dual output formats ("text" and "json") with corresponding transform mappings for both test execution and assertions.
• 🔧 Modified type definitions to include an optional format field, enabling developers to explicitly choose the desired output format for tests.

AI-generated content by prd may be incorrect


Pull Request Summary: Enhancing Prompt Testing

The push introduces several critical improvements for prompt-related testing:

  • New Interfaces: PromptTest now supports arrays of custom assertions, enabling detailed test validation. The addition of ContentSafetyOptions allows specifying both content safety rules and format preferences.

  • Format Specification: Users can now set output formats via the provided format option within the request context, defaulting to "text", aligning closely with public API expectations.

  • Error and Value Handling: Enhanced error messages improve developer understanding of failures. Added conditional checks ensure that only relevant formatting is applied per JSON content type when json is requested in ContentSafetyOptions.

These changes collectively make testing more robust and maintainable by extending assertion capabilities, improving format handling, and refining error communication through the updated API.

AI-generated content by pr-describe may be incorrect

Introduce 'format' to PromptTest and refactored transforms logic.
GenAIScript now defaults to extracting `text` but allows `format: "json"`.
@pelikhan pelikhan merged commit bc06cf3 into main Feb 8, 2025
15 checks passed
@pelikhan pelikhan deleted the testoutputtype branch February 8, 2025 17:22
@@ -156,13 +156,15 @@ scripts({

#### transform

By default, the `asserts` are executed on the raw LLM output.
However, you can use a javascript expression to select a part of the output to test.
By default, GenAIScript extracts the `text` field from the output before sending it to PromptFoo.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sentence structure could be improved for clarity. Consider rephrasing to avoid repetition and enhance readability.

AI-generated content by pr-docs-review-commit style may be incorrect

Copy link

github-actions bot commented Feb 8, 2025

LGTM with some notes:

  • The transform variable in test.ts is defined but never used
  • The format parameter in prompt_template.d.ts is documented but undefined usage
    This doesn't break anything but could be cleaned up for better maintainability.

Here are prrrogate fixes to remove unused variables:

--- a/packages/core/src/types/prompt_template.d.ts
+++ b/packages/core/src/types/prompt_template.d.ts
@@ -507,14 +507,10 @@
      * Additional deterministic assertions.
      */
     asserts?: PromptAssertion | PromptAssertion[]
 
-    /**
-     * Determines what kind of output is sent back to the test engine. Default is "text".
-     */
-    format?: "text" | "json"
-
+    /// Determines what kind of output is sent back to the test engine. Default is "text".
   }
 
 interface ContentSafetyOptions {

AI-generated content by pr-review may be incorrect

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant