Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add draft functionality for a generic sandboxed code running #580

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

plaguss
Copy link
Contributor

@plaguss plaguss commented Feb 21, 2025

Description

Work in progress

Add a generic code_runner function to allow running the code in the local environment, or in a sandbox, initially thought for e2b, but should be extendable to other cases.

Examples of calls from LiveCodeBench:

  • Call based functions (a function is generated from the LLM)
code = """\
import math
from typing import List

class Solution:
    def maxLength(nums: List[int]) -> int:
        max_len = 1  # At least one element is valid
        n = len(nums)
        for i in range(n):
            current_gcd = nums[i]
            current_lcm = nums[i]
            product = nums[i]
            # Check for single element
            if product == current_gcd * current_lcm:
                if 1 > max_len:
                    max_len = 1
            for j in range(i + 1, n):
                current_gcd = math.gcd(current_gcd, nums[j])
                current_lcm = (current_lcm * nums[j]) // math.gcd(current_lcm, nums[j])
                product *= nums[j]
                # Check if the condition holds
                if product == current_gcd * current_lcm:
                    current_length = j - i + 1
                    if current_length > max_len:
                        max_len = current_length
        return max_len
"""

verification_info = {'test_cases': [{'input': '[1, 2, 1, 2, 1, 1, 1]', 'output': '5', 'testtype': 'stdin'},
 {'input': '[2, 3, 4, 5, 6]', 'output': '3', 'testtype': 'stdin'},
 {'input': '[6,10]',
  'output': '2',
  'testtype': 'stdin'}], 'language': 'python'}

from lighteval.utils.code.runner import code_runner

test_cases = [test["input"] for test in verification_info["test_cases"]]
code = [code]
result = code_runner(test_cases, code, sandbox="e2b", template="call_based", fn_name="maxLength")
  • stdio based code (a script that prints to the console)
    ...

@HuggingFaceDocBuilderDev
Copy link
Collaborator

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants