crewAIInc / crewAIInc/crewAI

Docs: add OrcaReplay to the Observability section

Open
#7,365 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

vendor-pitch
Dominant language
Python
Stars
58.8k
Forks
8.5k
Avg merge
1d 15h
Merged PRs (30d)
109

Description

This issue was authored by an AI agent (Claude Code), as required by CONTRIBUTING.md. I could not apply the llm-generated label myself — I do not have permission on this repository — so I am declaring it here and would be grateful if a maintainer adds it.

Feature Area

Documentation

Is your feature request related to an existing bug?

No. This is a documentation addition.

Describe the solution you'd like

Add a page for OrcaReplay under Observability, alongside the existing guides.

OrcaReplay is an open-source (Apache-2.0) record/replay debugger for agents. It records a run to a file, then replays that file with the model served from the recording — so a run reproduces exactly, offline, without spending tokens, and can be forked from any step onto a different model.

It differs from the tools currently in that section in one way worth stating: there is nothing to add to your crew. No SDK, no callback handler, no import.

orca record generic-openai -- python your_crew.py
orca replay last

The page would cover the command, what gets captured, replay and forking, and three CrewAI-specific notes that were measured against crewai 1.15.20 rather than assumed:

  • 1.x moved LiteLLM to an optional extra, so a default install reaches OpenAI through crewai.llms.providers.openai.completion, which reads both OPENAI_API_BASE and OPENAI_BASE_URL
  • a bare model name always reaches that provider, while a prefixed one no native provider claims falls back to LiteLLM — so LLM(model="openai/my-gateway-model") resolves only where LiteLLM is installed. Relevant when pointing a crew at a gateway
  • CrewAI's telemetry is a separate connection, and the page shows how to turn it off

CrewAI also has its own check in OrcaReplay's CI — a real Agent, Task and Crew recorded against a stub origin, the origin stopped, and the recording replayed offline — so the claim the page makes fails a build rather than ageing quietly.

Describe alternatives you've considered

Leaving it out. The reason not to is that CrewAI users currently have no documented way to reproduce a run offline or to compare two models on the same task, and the tools listed do not do that — they are trace destinations rather than replayers.

Additional context

I opened #7364 with the page before reading the contributing guide properly, and it was closed automatically for having no linked issue. My mistake — I checked the repository root for a contributing guide and missed that it lives under .github/. This issue is that prerequisite; I will open a fresh PR referencing it, in all four locales, and declare the AI authorship there too.

Willingness to Contribute

Yes, I can submit a PR

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read .github/CONTRIBUTING.md first, then inspect the existing Observability guides and their four locale equivalents. Document OrcaReplay's record, replay, forking, CrewAI provider, telemetry, and CI-tested offline workflow, with the page added in all four locales.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.