crewAIInc / crewAIInc/crewAI

[BUG] kickoff_for_each_async / akickoff_for_each wipe replay data on every run

Open Beginner friendly
#6,704 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

no-issue-activity
Dominant language
Python
Stars
58.8k
Forks
8.5k
Avg merge
1d 15h
Merged PRs (30d)
109

Description

Description

Crew.kickoff_for_each_async and Crew.akickoff_for_each both delegate to the shared helper run_for_each_async in lib/crewai/src/crewai/crews/utils.py. After gathering results from all input copies, that helper unconditionally resets the shared replay store:

https://github.com/crewAIInc/crewAI/blob/main/lib/crewai/src/crewai/crews/utils.py#L500-L514

    results = await asyncio.gather(*async_tasks)

    total_usage_metrics = UsageMetrics()
    for crew_copy in crew_copies:
        if crew_copy.usage_metrics:
            total_usage_metrics.add_usage_metrics(crew_copy.usage_metrics)
    crew.usage_metrics = total_usage_metrics

    crew._task_output_handler.reset()
    return list(results)

Each crew copy already resets and repopulates the shared TaskOutputStorageHandler (backed by a single SQLite file) as part of its own kickoff_async/akickoff run. The crew._task_output_handler.reset() call above then deletes whatever the copies just persisted, immediately before returning.

This is the async/native-async counterpart of #6650 (which covers the synchronous kickoff_for_each); the fix for #6650 only touches Crew.kickoff_for_each in crew.py and does not touch run_for_each_async, so this path is still broken.

Steps to Reproduce

import asyncio
from unittest.mock import patch
from crewai import Agent, Crew, Task, Process
from crewai.tasks.task_output import TaskOutput

agent = Agent(role="Researcher", goal="Research a topic.", backstory="You are a careful researcher.")
task = Task(description="Research {topic}.", expected_output="A concise research note.", agent=agent)
crew = Crew(agents=[agent], tasks=[task], process=Process.sequential)

first_output = TaskOutput(description="Research first.", raw="first result", agent="Researcher")
latest_output = TaskOutput(description="Research latest.", raw="latest result", agent="Researcher")

async def main():
    with patch.object(Task, "execute_sync", side_effect=[first_output, latest_output]):
        results = await crew.kickoff_for_each_async(inputs=[{"topic": "first"}, {"topic": "latest"}])
    print("stored:", crew._task_output_handler.load())

asyncio.run(main())

Observed on unmodified main:

stored: []

Expected behavior

After kickoff_for_each_async (or akickoff_for_each) completes, crew._task_output_handler.load() should return the persisted output of at least one of the runs, so Crew.replay(task_id) has something to replay. Instead it is always empty.

Note: because the input copies for the async variants execute concurrently on separate threads against one shared SQLite-backed store, which copy's output "wins" is not deterministic the way it is for the sequential kickoff_for_each (where the last input in the list is guaranteed to be the one left standing). That ordering guarantee is a separate, harder problem; this issue is specifically about the unconditional reset silently discarding every run's output.

Possible Solution

Remove the trailing crew._task_output_handler.reset() call in run_for_each_async (lib/crewai/src/crewai/crews/utils.py), mirroring the fix applied to the synchronous path for #6650.

Environment

  • crewai: main branch (commit f15844b2)
  • Python 3.13

Additional context

This issue was found and this fix was prepared with AI assistance (Claude Code), per this repo's llm-generated labeling requirement in .github/CONTRIBUTING.md.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in lib/crewai/src/crewai/crews/utils.py at run_for_each_async, called by Crew.kickoff_for_each_async and Crew.akickoff_for_each, and inspect the trailing task-output reset after the gathered results are processed. Run the supplied reproduction with two inputs; done means the shared handler retains persisted output after completion so Crew.replay(task_id) has data.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sqlite
Domain
backend, databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.