Python Code ResultType Validation

[aristideubertas]

It would be very useful to have a built in validator for when we want our LLM to generate python code (or ideally any type of code, but in this case I am using python.

Currently I am using this approach:

from pydantic import BaseModel
from pydantic_ai import Agent, RunContext, ModelRetry

class CodeSnippet(BaseModel):
    code: str

# Initialize the agent with the desired model and result type
agent = Agent(
    model='openai:gpt-4o', 
    result_type=CodeSnippet,
    retries=0
)

# Implement the result validator to check for syntax errors
@agent.result_validator
async def validate_python_code(ctx: RunContext, result: CodeSnippet) -> CodeSnippet:
    try:
        # Attempt to compile the code to check for syntax errors
        compile(result.code, '<string>', 'exec')
    except SyntaxError as e:
        # Raise ModelRetry to prompt the agent to generate a new result
        raise ModelRetry(f'Syntax error in generated code: {e}') from e
    return result

# Use the agent to generate Python code based on a prompt
def main():
    prompt = 'Generate a Python function that adds two numbers, but introduce a syntax error.'
    result = agent.run_sync(prompt)
    print('Generated Python Code:')
    print(result.data.code)

if __name__ == '__main__':
    main()

Any ideas if this is something that you would like to implement?

[samuelcolvin]

we might support something like this in future, e.g. as a toolset, see #110.

But the Python should be run in an effective sandbox, not in the local process!

[Kludex]

Duplicated of #910.

Let's watch that issue. 🙏

[Kludex]

Ah wait, sorry, you want the code itself as a result? 🤔

Is there any LLM that sends that field back already?