huggingface / huggingface/smolagents

ENH: Parallel execution mode to overlap the code execution latency with the agent's generation process

Open
#2,248 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
29.3k
Forks
3k
Avg merge
17m
Merged PRs (30d)
2

Description

**Problem**
In `CodeAgent`, every step is strictly serial: the LLM streams the full `...` block first, smolagents then calls `parse_code_blobs(...)` and finally `python_executor(code_action)`. While the model is streaming tokens, the executor sits idle — even when the first top-level statements have already arrived and could legally start running. For steps that include slow tests, file I/O, network, or heavy data loads, that idle time is the dominant component of end-to-end latency.

To better understand this issue on smolagents, I run `CodeAgent` to solve this task `How many seconds would it take for a leopard at full speed to run through Pont des Arts`. The full run takes 42.5s wall-clock and decomposes into three steps. Step 1 emits a 2-line web_search call: ~24.7s of token generation followed by 1.88 s of execution (the web fetch). Step 2 is similarly shaped: ~4.9s gen + 1.57s exec. Step 3 emits a 5-line arithmetic combine: ~9.4s gen + 0.6ms exec. Total executor wall-time across the run is 3.45s (≈ 8.1 % of the end-to-end latency).

Our research team has proposed a solution to mitigate this issue, EAGER (Executing As you
GEneRate), https://arxiv.org/pdf/2604.00491. It studies this overlap systematically and reports substantial end-to-end latency reductions on execution-heavy code-generation workloads. We'd like to bring the same idea to smolagents as an opt-in.

**Proposed solution**
Add an opt-in flag (e.g. `eager: bool = False`) on `CodeAgent` that pipelines code generation with execution. The default behavior would be unchanged.

As tokens stream in, the agent watches the accumulating code buffer, identifies top-level statements as soon as they parse cleanly, and starts running them in order while later tokens are still being produced.
The executor's persistent state (variables, imports, function definitions) is reused across these dispatched chunks, so values flow between them naturally; the per-step result — concatenated logs, last-chunk output, is_final_answer — is assembled at the end exactly as today. The mechanism sits between the streaming model output and the existing python_executor interface, so it should compose with LocalPythonExecutor, E2BExecutor, and any future backend that exposes the same per-call shape.

Trade-offs I'm aware of:
- Lightweight statements don't pay startup cost of the executor. Eager only dispatches a chunk when the accumulated executable code is non-trivial; trivially short top-level statements are coalesced and run together with the next chunk (or with the final block at end of step). This avoids the "many tiny exec calls" pathology and keeps overhead bounded for code-light steps.
- Some overhead from the watching (mainly from the AST parsing) during streaming. When the flag is on, the only added work on the streaming path is incremental AST parsing of the growing code buffer. Parsing only runs when the buffer changes; incremental parsing on partial Python is cheap, and if profiling ever shows it on the hot path, batching the parse across multiple tokens is a straightforward fallback. I do not expect any visible impact on the user-facing token stream.
- Off by default, so existing users are unaffected.

I'm a PhD student working on AI for software engineering and would commit to maintaining this on an ongoing basis (responding to issues, adapting to API changes). If you are interested in this feature, I'll be more than happy to submit a PR.

**Is this not possible with the current options.**
Not currently.

**Alternatives considered**
Implementing this entirely outside smolagents as a third-party wrapper. This is feasible but would have to monkey-patch _step_stream and the parsing path. A first-class opt-in flag is much cleaner for users and easier to keep correct as the agent loop evolves.

**Additional context**
Paper: https://arxiv.org/pdf/2604.00491
Personal Homepage: https://v587su.github.io/

---

### Checklist
- [x] I have searched the existing issues and have not found a similar feature request.
- [x] I have verified that this feature is not already implemented in the latest version.
- [x] I am willing to work on this feature and submit a pull request. (optional)

Contributor guide

Open the contributing guide

Research direction

Start by reading CodeAgent's _step_stream and the existing parse_code_blobs(...) to python_executor(...) flow. Compare current behavior using the described CodeAgent leopard task. Done means an opt-in eager flag preserves default behavior while ordered chunks execute during generation, reuse executor state, and assemble the same per-step results across supported executors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai-infra-agents
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.