get2knowio / get2knowio/maverick
Add OpenInference tracing layer for LLM observability
- Dominant language
- Python
- Stars
- 4
- Forks
- 0
- Avg merge
- 17h 37m
- Merged PRs (30d)
- 7
Description
## Summary
Add an optional OpenInference/OpenTelemetry tracing layer that auto-captures Claude API call details and models Maverick's workflow/agent execution as hierarchical spans, enabling deep LLM observability alongside the existing session journal.
## Motivation
Maverick's `SessionJournal` captures a rich flat stream of 20+ workflow event types in JSONL format. This is excellent for workflow-level replay and debugging, but has gaps for **LLM-level observability**:
- **No automatic capture** of Claude API call internals (invocation parameters, cache read/write token breakdowns, raw tool call structure, streaming chunk details)
- **Flat event stream** — no hierarchical parent/child relationships between workflow phases, agent executions, and individual LLM calls
- **No ecosystem connectivity** — session logs require custom tooling to analyze; they can't flow to standard observability backends
[OpenInference](https://github.com/Arize-ai/openinference) is an Apache 2.0 specification by Arize AI that extends OpenTelemetry with AI-specific semantic conventions (10 span kinds: LLM, AGENT, TOOL, CHAIN, etc.). It provides:
- **`openinference-instrumentation-anthropic`**: Auto-instruments all Anthropic SDK calls (Messages API, streaming, tool use) with zero application code changes
- **`openinference-instrumentation-mcp`**: Auto-instruments MCP protocol interactions
- **Standard OTel wire format**: Traces flow to Phoenix (local), Langfuse, Grafana Tempo, Jaeger, or any OTLP-compatible backend
- **844 GitHub stars**, 31+ Python instrumentation packages, active daily releases
### Relationship to existing work
- **Complements SessionJournal** — does NOT replace it. SessionJournal captures Maverick's custom workflow events (preflight, validation, rollbacks, loop iterations). OpenInference captures LLM call details that SessionJournal does not.
- **Builds on #18** (trace ID correlation) — the `trace_id` from #18 should be used as the OTel trace ID, unifying both systems under one correlation key.
- **Prerequisite for #17** (agent output evaluation) — structured traces are the foundation for systematic evaluation via Phoenix datasets and experiments.
### Key gap: No Claude Agent SDK instrumentation
There is no `openinference-instrumentation-claude-agent-sdk` package. The Anthropic instrumentor covers the raw `anthropic` Python SDK (HTTP client), not the higher-level agent SDK. This means:
- LLM calls made through `ClaudeSDKClient` / `query()` will be auto-captured (the Anthropic instrumentor patches `Messages.create` / `AsyncMessages.create`)
- Agent-level structure (which agent, which workflow phase, which step) must be **manually instrumented** with custom spans
### OTel GenAI convergence note
OpenTelemetry's own [GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) (using `gen_ai.*` attributes) are in Development status and may eventually supersede OpenInference's `llm.*` namespace. To mitigate this:
- Keep OpenInference-specific attribute names out of Maverick's core domain code
- Isolate all span creation behind a thin internal API that could be retargeted later
- Monitor the OTel GenAI working group's [agent span spec](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/)
## Proposed Architecture
### New module structure
```
src/maverick/
├── tracing/
│ ├── __init__.py # Public API: setup_tracing(), teardown_tracing()
│ ├── config.py # TracingConfig (Pydantic model)
│ ├── provider.py # OTel TracerProvider setup, exporter config
│ ├── spans.py # Helpers for creating workflow/agent/tool spans
│ └── instrumentation.py # Auto-instrumentation orchestration
```
### Configuration
```yaml
# maverick.yaml or ~/.config/maverick/config.yaml
tracing:
enabled: false # Opt-in
exporter: otlp_http # otlp_http | otlp_grpc | console
endpoint: "http://localhost:6006/v1/traces" # Phoenix default
project_name: maverick
include_inputs: true # Privacy: hide prompt content
include_outputs: true # Privacy: hide response content
```
CLI override:
```bash
maverick fly feature --tracing --tracing-endpoint http://localhost:6006/v1/traces
```
### Auto-instrumentation (zero-code)
On startup (when tracing is enabled):
```python
from openinference.instrumentation.anthropic import AnthropicInstrumentor
AnthropicInstrumentor().instrument(tracer_provider=tracer_provider)
```
This automatically captures for every Claude API call:
- Model name and invocation parameters (temperature, max_tokens, etc.)
- Full input/output messages (configurable via `TraceConfig`)
- Token counts (prompt, completion, cache read, cache write)
- Tool definitions and tool call/result pairs
- Streaming chunk details
### Manual workflow spans
Add explicit spans around Maverick's execution layers to create a meaningful hierarchy:
```
[CHAIN] workflow: feature (trace_id from #18)
├── [CHAIN] phase: preflight
│ ├── [CHAIN] check: git_clean
│ └── [CHAIN] check: github_auth
├── [AGENT] step: implement (agent=ImplementerAgent)
│ ├── [LLM] anthropic.messages.create ← auto-captured
│ ├── [TOOL] mcp: read_file ← auto-captured if MCP instrumented
│ └── [LLM] anthropic.messages.create ← auto-captured
├── [CHAIN] step: validate (type=python)
└── [AGENT] step: review (agent=CodeReviewerAgent)
└── [LLM] anthropic.messages.create ← auto-captured
```
Integration points:
- `WorkflowFileExecutor.execute()` — root CHAIN span with workflow metadata
- `_execute_step()` — AGENT or CHAIN span per step (type-dependent)
- `MaverickAgent._execute()` — AGENT span with agent name, model, tool permissions
- Use `using_session(session_id)` context manager to correlate with session journal
### Dependencies
New packages:
- `openinference-semantic-conventions` — attribute name constants
- `openinference-instrumentation` — core utilities, `TraceConfig`, context managers
- `openinference-instrumentation-anthropic` — Anthropic auto-instrumentation
- `opentelemetry-sdk` — TracerProvider, span processors
- `opentelemetry-exporter-otlp-proto-http` — OTLP HTTP exporter
Optional:
- `openinference-instrumentation-mcp` — MCP auto-instrumentation (evaluate separately)
- `arize-phoenix` — local Phoenix instance (dev/evaluation only, not a runtime dependency)
## Scope
### In scope
- `TracingConfig` Pydantic model with config file + CLI support
- OTel TracerProvider setup with configurable OTLP exporter
- Anthropic auto-instrumentation toggle
- Manual span creation for workflow phases, steps, and agent executions
- `session.id` propagation via `using_session()` context manager
- Correlation with #18 trace IDs (if implemented) or independent UUID
- Graceful no-op when tracing is disabled (zero overhead)
- Tests for span creation, config parsing, provider lifecycle
### Out of scope
- Phoenix server management (users run `phoenix launch` themselves)
- MCP auto-instrumentation (evaluate in a follow-up)
- Evaluation framework integration (see separate issue linking to #17)
- Custom OTel collector configuration
- Multi-backend tracing (only Claude/Anthropic for now)
## Acceptance Criteria
- [ ] `TracingConfig` Pydantic model with validation
- [ ] `setup_tracing()` / `teardown_tracing()` lifecycle API
- [ ] Anthropic auto-instrumentation active when tracing enabled
- [ ] Manual CHAIN spans for workflow execution and phases
- [ ] Manual AGENT spans for agent step execution
- [ ] `session.id` set on all spans within a workflow run
- [ ] Privacy controls: configurable input/output hiding
- [ ] Zero overhead when `tracing.enabled: false`
- [ ] `--tracing` CLI flag on `fly` and `workflow run` commands
- [ ] Tests for provider setup, span creation, config parsing
- [ ] No impact on existing SessionJournal behavior
- [ ] Documentation: setup guide with Phoenix quickstart
## References
- [OpenInference GitHub](https://github.com/Arize-ai/openinference) (844 stars, Apache 2.0)
- [OpenInference semantic conventions](https://arize-ai.github.io/openinference/spec/semantic_conventions.html)
- [openinference-instrumentation-anthropic](https://pypi.org/project/openinference-instrumentation-anthropic/)
- [Arize Phoenix](https://github.com/Arize-ai/phoenix) (8.5K stars)
- [OTel GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) (Development status)
- [OTel GenAI agent spans](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/)
- Related: #18 (trace ID correlation), #17 (agent output evaluation)
- Current session logging: `src/maverick/session_journal.py`
Contributor guide
Assessment
This issue has not been assessed yet.