support basic text output in tests #1113
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduce 'format' to PromptTest and refactored transforms logic.
• 📝 Updated test documentation to clarify that GenAIScript now extracts the "text" field by default, with an option to use raw JSON output by specifying format: "json".
• ⚙️ Enhanced test configuration logic to support dual output formats ("text" and "json") with corresponding transform mappings for both test execution and assertions.
• 🔧 Modified type definitions to include an optional format field, enabling developers to explicitly choose the desired output format for tests.
Pull Request Summary: Enhancing Prompt Testing
The push introduces several critical improvements for prompt-related testing:
New Interfaces:
PromptTest
now supports arrays of custom assertions, enabling detailed test validation. The addition ofContentSafetyOptions
allows specifying both content safety rules and format preferences.Format Specification: Users can now set output formats via the provided
format
option within the request context, defaulting to "text", aligning closely with public API expectations.Error and Value Handling: Enhanced error messages improve developer understanding of failures. Added conditional checks ensure that only relevant formatting is applied per JSON content type when
json
is requested inContentSafetyOptions
.These changes collectively make testing more robust and maintainable by extending assertion capabilities, improving format handling, and refining error communication through the updated API.