crewAIInc / crewAIInc/crewAI

[BUG] kickoff_for_each clears latest replay records before returning

Open Beginner friendly
#6,650 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
58.8k
Forks
8.5k
Avg merge
1d 15h
Merged PRs (30d)
109

Description

Description

The documented synchronous, non-streaming Crew.kickoff_for_each replay contract is not preserved at current main / CrewAI 1.15.6 (b3aaaab023a53a08db4d36c9430b3463f1efcc7d).

The replay guide says kickoff_for_each supports replay from the most recent crew run:

Each copied kickoff already resets the shared latest-output store before executing and then persists its records. Because synchronous kickoff_for_each runs the copies sequentially, the final copy leaves the valid latest run. The parent method then performs another self._task_output_handler.reset() immediately before returning, deleting those records.

A deterministic regression on unmodified source receives both expected batch outputs but observes [] in the replay store after return. An ordinary-kickoff replay control passes.

Steps to Reproduce

  1. Pin the repository and create its Python 3.12 environment:

    git clone https://github.com/crewAIInc/crewAI.git
    git -C crewAI checkout --detach b3aaaab023a53a08db4d36c9430b3463f1efcc7d
    cd crewAI
    uv sync --python 3.12 --frozen --group dev
    
  2. Copy test_kickoff_for_each_replay.py from the public evidence repository into lib/crewai/tests/.

  3. Run the regression on unmodified production code:

    ./.venv/Scripts/python.exe -m pytest -q -n 0 --tb=short --disable-warnings \
      lib/crewai/tests/test_kickoff_for_each_replay.py::test_kickoff_for_each_persists_only_the_most_recent_run_for_replay
    
  4. Run the ordinary-kickoff control:

    ./.venv/Scripts/python.exe -m pytest -q -n 0 --tb=short --disable-warnings \
      lib/crewai/tests/test_kickoff_for_each_replay.py::test_regular_kickoff_persists_and_replays_latest_run
    

Observed on unmodified source:

  • regression: 1 failed; len(stored) is 0, expected 1;
  • ordinary-kickoff control: 1 passed;
  • combined module: 1 failed, 1 passed.

Expected behavior

After one synchronous kickoff_for_each call over inputs first and latest:

  • both batch outputs should return;
  • the replay store should contain exactly the final run with {"topic": "latest"};
  • Crew.replay(final_task_id) without an input override should execute again using the final stored input.

Screenshots/Code snippets

Failing assertion on the unmodified pinned source:

assert len(stored) == 1
E       assert 0 == 1
E        +  where 0 = len([])

The root-cause-aligned candidate removes only the post-loop reset:

         if not self.stream:
             self.usage_metrics = total_usage_metrics
-        self._task_output_handler.reset()
         return results

With that deletion, the strengthened test verifies:

  • final stored inputs;
  • a third Task.execute_sync call during replay;
  • the final interpolated description and returned output;
  • a truthy persisted replay marker.

Operating System

Windows 11

Python Version

3.12.10

crewAI Version

1.15.6

crewAI Tools Version

1.15.6 (installed by the locked workspace; not involved in this test)

Virtual Environment

Venv (uv sync --python 3.12 --frozen --group dev)

Evidence

Public evidence repository:

https://github.com/HarperZ9/crewai-kickoff-replay-receipt

Read-back verification at commit cfe410c8cdce4e999bfeb9938193a45bfc9f22c8:

  • public bundle verifier: MATCH, 362 checks;
  • canonical result SHA-256: 2c9a17d1c6f049cecb8ef10917d4369ff68e626011f8fe222d329407d21e63aa;
  • Project Telos action receipt SHA-256: e5b000658a704f1e099296a3005ddc13a8f3e6f620554c3c23365a7e38062aec;
  • candidate regression/control/combined: 1/1, 1/1, 2/2;
  • selected existing kickoff_for_each tests: 8/8;
  • selected existing replay tests: 9/9;
  • Ruff, formatting, git diff --check, and reverse patch checks: pass.

The test substitutes Task.execute_sync with deterministic TaskOutput generation, so it exercises Crew copying, interpolation, SQLite persistence, stored-input recovery, and replay without a model call. Repository network blocking remains active on the exercised Python paths, with exact loopback hosts allowed for Windows socket.socketpair.

Possible Solution

Remove the terminal task-output reset from synchronous kickoff_for_each and add the focused regression/control coverage in the evidence repository.

I would be glad to submit the one-line production patch and tests if maintainers want this behavior.

Additional context

  • Scope is synchronous, non-streaming, one-task execution. Async, streaming, multi-task, and real-model behavior were not validated.
  • The selected nearby suites passed; the full repository suite was not run.
  • Removing the reset means kickoff_for_each([]) preserves the previous replay snapshot instead of clearing it. That empty-input contract should be an explicit maintainer decision.
  • RED chronology is recorded but not cryptographically time-sealed beside an unmodified-source hash.
  • This issue was prepared with AI assistance under human authorization. Per .github/CONTRIBUTING.md, it must carry the llm-generated label; I am requesting/applying that label explicitly.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read the synchronous kickoff_for_each implementation in lib/crewai/src/crewai/crew.py, especially the post-loop task-output reset near lines 1106-1110. Copy test_kickoff_for_each_replay.py into lib/crewai/tests/ and run the two focused pytest commands from the issue. Done means both batch outputs return and the final replay record remains available for Crew.replay without an input override.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sqlite
Domain
backend, databases, testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
87/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.