traceloop / traceloop/openllmetry

πŸš€ Feature: CrewAI: instrument BaseTool.run to emit tool-execution spans

Open
#4,452 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.4k
Forks
1.1k
Avg merge
8d 14h
Merged PRs (30d)
2

Description

Which component is this feature for?

All Packages

πŸ”– Feature description

Component: CrewAI Instrumentation β€” not in the dropdown above, so I selected "All Packages."

opentelemetry-instrumentation-crewai==0.62.3 creates no span for tool execution. A trace consumer can see which tools an agent had, but not which ran, in what order, with what arguments, or what they returned.

CrewAIInstrumentor._instrument wraps four call sites, and _uninstrument mirrors them β€” instrumentation.py#L87-L94: Crew.kickoff, Agent.execute_task, Task.execute_sync, LLM.call. crewai.tools.base_tool BaseTool.run is not among them.

Tools reach the trace only through _serialize_tools, which filters each tool to name and description β€” crewai_span_attributes.py#L161-L177 β€” landing as crewai.agent.tools and crewai.task.tools.

🎀 Why is this feature needed ?

Tool-level visibility is the part of an agent trace that shows what the agent actually did. Without a span per tool execution there's no tool latency, no tool error surface (a raising tool appears only as the agent span's status), and no record of the arguments a tool was called with.

Verified live β€” crewai==1.15.18, traceloop-sdk==0.62.3, opentelemetry-instrumentation-crewai==0.62.3, Python 3.12.3. A two-tool crew (fetch_ticket, send_email) where both tools demonstrably executed; the full span inventory for the run:

   3  openai.chat
   1  Support Responder.agent
   1  <task description>.task
   1  crewai.workflow

That's a name-agnostic inventory of every span emitted, so it holds regardless of what a tool span would have been called. Nothing corresponds to a tool invocation.

The existing tool attributes don't compensate, because they're snapshotted at span start via CrewAISpanAttributes(span, instance) before the tools run. After both tools completed successfully, the exported spans still read:

crewai.agent.tools_results = []
crewai.task.used_tools     = 0
crewai.task.tools_errors   = 0

So the counters that exist are also wrong β€” arguably a separate defect, but I mention it here because it's the obvious answer to "isn't the tool data already on the agent span?"

Your own RFC #3460 already specifies this span:

Description: Execution of a tool, function, or capability by an agent.

Span Kind: CLIENT

Required Attributes:

Attribute Type Description Examples
gen_ai.tool.name string Tool/function name "web_search", "calculator", "read_file"
gen_ai.tool.type string Tool category "api", "function", "code", "mcp", "native", "browser"
gen_ai.operation.name string Operation performed "execute", "invoke", "call"

Its framework examples read "All frameworks: Function calling, custom tools, built-in tools", then name Agno, Haystack and Smolagents. CrewAI isn't named, which is why this is a feature request against your documented direction rather than a spec-violation bug.

✌️ How do you aim to achieve this?

Wrap crewai.tools.base_tool BaseTool.run alongside the existing wraps, emitting one span per tool invocation with gen_ai.tool.name, gen_ai.tool.type and gen_ai.operation.name, plus arguments and result where content capture is enabled.

I'd default to the RFC #3460 shape (gen_ai.tool.execute, CLIENT) since it's your documented direction, unless you'd rather track the upstream OTel GenAI execute_tool span β€” specified at https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md (note the GenAI conventions moved out of the main semantic-conventions repo). Happy to write it either way; just say which before I start.

πŸ”„οΈ Additional Information

Two things I want to pre-empt, since I checked both:

  1. traceloop-sdk does ship @tool / @atool, which create a {name}.tool span and do set gen_ai.tool.name β€” decorators/base.py#L203-L211. That's manual instrumentation of my own functions; it doesn't hook CrewAI's tool execution, so it can't cover tools CrewAI invokes internally. This request is about the auto-instrumentation path.
  2. This isn't a general semconv complaint. The agent and LLM spans use gen_ai.* extensively β€” GEN_AI_PROVIDER_NAME, GEN_AI_OPERATION_NAME, GEN_AI_AGENT_NAME, GEN_AI_INPUT_MESSAGES/OUTPUT_MESSAGES β€” and the package ships test_semconv_compliance.py. The gap is specifically tool execution.

Possibly related: #4397 lists CrewAI among the unwired packages in the conformance contract, pending cassettes. Glad to help wire a CrewAI cassette β€” I have a working harness.

Design note for whoever implements this: gen_ai.tool.call.id won't be available on CrewAI regardless, since CrewAI doesn't surface the model's tool call id to BaseTool.run (crewAIInc/crewAI#7178, proposed fix in PR #7189 β€” open, not merged).

πŸ‘€ Have you spent some time to check if this feature request has been raised before?
  • I checked and didn't find similar issue
Are you willing to submit PR?

Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up β€” it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in packages/opentelemetry-instrumentation-crewai/opentelemetry/instrumentation/crewai/instrumentation.py, where Crew.kickoff, Agent.execute_task, Task.execute_sync, and LLM.call are wrapped, and inspect crewai_span_attributes.py for existing tool data. Compare the requested BaseTool.run span with RFC #3460 and the upstream execute_tool convention before choosing the span shape. Update the relevant instrumentation tests, including test_semconv_compliance.py, so tool invocations expose the agreed attributes and content capture behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.