huggingface / huggingface/smolagents

BUG: Syntax errors cause print statements from previous step to emit.

Open
#1,998 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
29.4k
Forks
3k
Avg merge
17m
Merged PRs (30d)
2

Description

Problem
This is a very minor and specific/rare bug, but if text printing occurs in step 1 and a SyntaxError error occurs in step 2, then the print outputs in step 1 will emit in step 2.

Steps to reproduce

from smolagents import CodeAgent

agent = CodeAgent(tools=[], model=model)

response = agent.run(
    "Let's test your ability to handle syntax errors. In two separate steps, do the following:\n"
    "1. print('This is placeholder text. The real test will happen in step 2.')\n"
    "2. print('Starting Test:'), and then write code that has a syntax error. Don't use eval, and do not catch the exception. Just emit code with wrong syntax in the code block.\n"
    "After you've done both steps and they've raised an exception, stop and return via `final_answer`."
)

Actual behavior and error logs

╭──────────────────────────────────────────────────── New run ────────────────────────────────────────────────────╮
│                                                                                                                 │
│ Let's test your ability to handle syntax errors. In two separate steps, do the following:                       │
│ 1. print('This is placeholder text. The real test will happen in step 2.')                                      │
│ 2. print('Starting Test:'), and then write code that has a syntax error. Don't use eval, and do not catch the   │
│ exception. Just emit code with wrong syntax in the code block.                                                  │
│ After you've done both steps and they've raised an exception, stop and return via `final_answer`.               │
│                                                                                                                 │
╰─ OpenAIModel - gpt-5 ───────────────────────────────────────────────────────────────────────────────────────────╯
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Step 1 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 ─ Executing parsed code: ──────────────────────────────────────────────────────────────────────────────────────── 
  print('This is placeholder text. The real test will happen in step 2.')                                          
 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────── 
Execution logs:
This is placeholder text. The real test will happen in step 2.

Out: None
[Step 1: Duration 39.86 seconds| Input tokens: 2,064 | Output tokens: 2,139]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Step 2 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 ─ Executing parsed code: ──────────────────────────────────────────────────────────────────────────────────────── 
  print('Starting Test:')                                                                                          
  def bad_syntax(                                                                                                  
      pass                                                                                                         
 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────── 
Execution logs:
This is placeholder text. The real test will happen in step 2.

Code parsing failed on line 2 due to: SyntaxError: '(' was never closed (<unknown>, line 2)
def bad_syntax(               ^
[Step 2: Duration 14.23 seconds| Input tokens: 4,264 | Output tokens: 4,192]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Step 3 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 ─ Executing parsed code: ──────────────────────────────────────────────────────────────────────────────────────── 
  final_answer("Completed: Step 1 printed placeholder text. Step 2 printed 'Starting Test:' and intentionally      
  raised a SyntaxError. Stopping as requested.")                                                                   
 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────── 
Final answer: Completed: Step 1 printed placeholder text. Step 2 printed 'Starting Test:' and intentionally raised 
a SyntaxError. Stopping as requested.

Potential Fix
I believe this is caused by evaluate_python_code in local_python_executor.py:

    try:
        expression = ast.parse(code)
    except SyntaxError as e:
        raise InterpreterError(
            f"Code parsing failed on line {e.lineno} due to: {type(e).__name__}: {str(e)}\n"
            f"{e.text}"
            f"{' ' * (e.offset or 0)}^"
        )

    if state is None:
        state = {}
    static_tools = static_tools.copy() if static_tools is not None else {}
    custom_tools = custom_tools if custom_tools is not None else {}
    state["_print_outputs"] = PrintContainer()
    state["_operations_count"] = {"counter": 0}

You don't get a new PrintContainer if a SyntaxError occurs, so the PrintContainer from the previous step leaks. This can be solved by reversing the order of these two code chunks (i.e. the state manipulation code occuring before ast.parse).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in local_python_executor.py at evaluate_python_code, then reproduce the two-step execution described in the issue. Verify that a syntax error in the second step does not emit print output from the first step, while the syntax error is still reported as expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.