run_tasks creates one pending task per input despite the parallelism limit

Open
#1,418 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
68/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
docker, python
Domain
backend

Research direction

Start in nemo_platform/beta/evaluator/agent_eval/runtimes/docker_sandbox.py at DockerSandboxAgentRuntime.run_tasks(), lines 113-128, and run the attached run.py reproduction. Done means configured parallelism limits both active sandbox work and admitted pending child tasks, rather than creating one pending task per input.

Written by the indexing model from the issue text.

Description

bug

Description

DockerSandboxAgentRuntime.run_tasks() limits the number of sandbox tasks executing concurrently according to the configured parallelism, but creates a child asyncio task for every input task before waiting for them to complete.

As a result, parallelism bounds active sandbox work but does not bound the number of admitted pending tasks.

I reproduced this behavior with 512 input tasks and parallelism=16.

Observed behavior:

Input tasks: 512
Configured parallelism: 16
Peak active sandbox tasks: 16
Pending child tasks before release: 512
External service contacted: No

The number of pending asyncio tasks therefore scales with the total input size rather than the configured parallelism. Large evaluation batches can consequently create substantial numbers of pending tasks even when active sandbox execution is intentionally limited.

Affected Version

Confirmed on:

nemo-platform-plugin==0.3.0
DockerSandboxAgentRuntime.run_tasks()
nemo_platform/beta/evaluator/agent_eval/runtimes/docker_sandbox.py:113-128

PoC runtime:

Python 3.11.0rc1

Other versions have not been tested.

Reproduction

The attached PoC loads the released DockerSandboxAgentRuntime.run_tasks() implementation from the 0.3.0 wheel and supplies 512 tasks with parallelism set to 16.

The per-task operation is replaced with a controlled local implementation that holds active work pending long enough to measure both execution concurrency and the number of child tasks already created.

Observed result:

input tasks: 512
configured parallelism: 16
peak active sandbox tasks: 16
pending child tasks before release: 512
input-proportional pending task creation reproduced: yes

No external service is contacted during reproduction.

Attachment

run.py

Dominant language
Python
Stars
79
Forks
24
Avg merge
1d 14h
Merged PRs (30d)
578

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA-NeMo/nemo-platform

All issues in NVIDIA-NeMo/nemo-platform

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.