CallableAgentTaskRunner.run_tasks eagerly creates one task per trial despite parallelism limit

Description

CallableAgentTaskRunner.run_tasks() limits concurrent agent execution using its configured parallelism value.

However, the complete set of per-trial operations is still scheduled at once. The concurrency control limits how many agent calls execute simultaneously, but it does not bound the number of outstanding child tasks created for the input set.

I reproduced this behavior with 512 controlled tasks and parallelism set to 8.

Observed behavior:

Input tasks: 512
Configured parallelism: 8
Peak active agent calls: 8
Pending child tasks before release: 512
External service contacted: No

The execution limit therefore works as intended, but task admission still grows directly with the total number of trials.

For large evaluation runs, this creates avoidable task-management and memory overhead compared with keeping only a bounded amount of work outstanding.

Affected Version

Confirmed on:

nemo-platform-plugin==0.3.0
Python 3.13.14
nemo_platform/beta/evaluator/agent_eval/runtimes/callable_runtime.py:56-73
CallableAgentTaskRunner.run_tasks()

Other versions have not been tested.

Reproduction

The attached PoC executes the released CallableAgentTaskRunner.run_tasks() implementation with 512 controlled tasks and parallelism set to 8.

The agent callable is replaced with a local asynchronous operation that records active calls and waits on a shared gate. This allows the harness to observe both the active-agent concurrency limit and the number of child tasks created before any agent call completes.

Observed result:

input tasks: 512
configured parallelism: 8
peak active agent calls: 8
pending child tasks before release: 512
task-admission behavior reproduced: yes

Only eight agent calls execute concurrently, but all 512 per-task child operations are already pending.

No agent service or other external endpoint is contacted during reproduction.

Expected behavior

If the configured parallelism is intended to provide bounded execution for large evaluation runs, the number of outstanding per-trial tasks should also remain bounded rather than growing directly with the total number of input tasks.

A bounded worker pool, queue, or batched scheduling strategy could preserve the existing parallelism limit without eagerly creating one task for every trial.

Attachment

run.py

Start with nemo_platform/beta/evaluator/agent_eval/runtimes/callable_runtime.py:56-73 and inspect CallableAgentTaskRunner.run_tasks(). Run the attached run.py reproduction to observe active calls and pending child tasks. Done means large inputs no longer create one outstanding task per trial while the configured parallelism limit remains intact.