a2aproject / a2aproject/a2a-java

[Bug]: some observability capabilities loss on A2A

Abierto
#1,132 5 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Java
Estrellas
490
Forks
172
Merge medio
1 d 6 h
PR fusionados (30 d)
55

Descripción

### What happened?

First, I would like to confirm whether the community has a roadmap for observability capabilities and whether the current capabilities are considered complete by the community.
Below are some issues I encountered when building observability capabilities based on a2a-java.

---

## Issue 1: Server-side SSE per-event spans are missing

### Summary

When a Server Agent sends SSE streaming events back to the Client, there are **no per-event spans on the Server side**. The Server span ends as soon as the stream publisher is created (`"Stream publisher created"`), making the actual event emission process a complete black box.

### Current Behavior

```
Client Agent Server Agent
───────── ─────────
[CLIENT Span: sendMessage] ──HTTP──→ [SERVER Span: onMessageSendStream]
│ │
│ Span ends after stream init │ Span ends after publisher returns
▼ ▼
[CLIENT Span: event-1] ❌ No Span
[CLIENT Span: event-2] ❌ No Span
[CLIENT Span: event-3] ❌ No Span
│ │
▼ ▼
Link to initial Event sending completely
CLIENT Span (non-parent-child) untraceable
```

- The Server span closes before any SSE events are actually sent.
- The Client side has per-event spans (linked via `addLink`), but the Server side has nothing.
- Client and Server event spans cannot be correlated one-to-one.

### Expected Behavior

Each SSE event sent by the Server should have an `INTERNAL` span as a child of the Server span (via context restoration), with attributes such as `streaming.event.kind` (status / artifact / completed / error). This would close the end-to-end trace loop.

### Impact

- Server-side event emission is a black box — no visibility into which events were sent, when, or how long each took.
- End-to-end trace tree is broken: the Server span has no children, and Client event spans are only linked (not parent-child), so they don't appear in the trace tree view.
- Cannot correlate Client-side event spans with Server-side event sending.

---

## Issue 2: Push notification delivery is completely untraceable

### Summary

Push notification delivery has **no tracing at all**. The active push callback from the Server Agent to the Client's webhook endpoint cannot be correlated to the original request.

### Current Behavior

- Push notification configuration management (set/get/delete webhook) has spans.
- But the actual **push notification delivery** (the webhook callback) has zero tracing.
- There is no span, no trace context propagation, and no way to link the push delivery back to the original task/request.

### Expected Behavior

- Push notification delivery should create a span with attributes such as `push.notification.url`, `notification.topic`, and `streaming.event.kind`.
- The delivery span should be linkable to the original request span (via stored context or span link).

### Impact

- Cannot track whether push notifications were successfully delivered.
- Cannot measure push notification delivery latency.
- Cannot correlate push notifications back to the original task or subscription.

---

## Issue 3: No business-level semantic attributes on spans

### Summary

A2A protocol-layer spans carry basic attributes (operation name, task/message/context ID), but there are **no business-level semantic attributes** such as task type,.

### Current Behavior
no genai attributes, no business attributes.

### Expected Behavior

The SDK should provide a mechanism (e.g., callback, annotation, or API) for users to enrich spans with these business-level attributes.

### Impact

- Traces show "what happened at the protocol level" but not "what happened at the business level."

---

## Issue 4: Task state transitions are not tracked

### Summary

Task status attributes (e.g., `submitted` → `working` → `completed`) are **reserved in the code but not activated**. Task state transitions are invisible in traces.

### Current Behavior

- The span attribute for task status appears to be defined but never populated with actual values.
- When a task transitions from `submitted` to `working` to `completed`, no span attribute records this transition.

### Expected Behavior

Each SSE status event should carry a `task.status` attribute reflecting the current task state, enabling state-machine visualization and duration analysis per state.

### Impact

- Cannot track task progress through its lifecycle.
- Cannot measure time spent in each task state.
- Cannot build task state-machine visualizations in observability dashboards.

---

## Issue 5: No Metrics implementation in core module

### Summary

The a2a-java core module has **zero Metrics implementation**. There are no histograms, counters, or any other instruments measuring operation duration, throughput, or token usage.

### Current Behavior

| Expected Metric | Type | Status |
| --- | --- | --- |
| `gen_ai.client.operation.duration` | Histogram | ❌ Missing |
| `gen_ai.client.token.usage` | Histogram | ❌ Missing |
| `a2a.task.request.duration` | Histogram | ❌ Missing |

- There is no `gen_ai.client.operation.duration` metric, which is the standard OTel GenAI operation duration histogram.

### Expected Behavior

The core SDK module should provide built-in metrics instruments, at minimum:

- `gen_ai.client.operation.duration` (OTel GenAI standard) — operation latency histogram
- `gen_ai.client.token.usage` (OTel GenAI standard) — token usage histogram by `gen_ai.token.type`
- Task request duration covering the full SSE streaming lifecycle (from stream initiation to last event)

### Impact

- Cannot measure operation latency or throughput at scale.
- Cannot set up SLO/SLA alerts based on latency percentiles.
- Cannot quantify agent interaction performance trends.
- Token usage is not tracked, making cost analysis impossible.

---

## Issue 6: Span attributes use custom `gen_ai.agent.a2a.*` prefix instead of official `gen_ai.*` SemConv

### Summary

All A2A span attributes use a custom `gen_ai.agent.a2a.*` prefix (e.g., `gen_ai.agent.a2a.operation.name`, `gen_ai.agent.a2a.context_id`) instead of the official OpenTelemetry GenAI Semantic Conventions (`gen_ai.*`). This prevents automatic integration with OTel GenAI dashboards and alerting rules.

### Current Behavior

| Official GenAI SemConv Attribute | a2a-java Attribute | Status |
| --- | --- | --- |
| `gen_ai.provider.name` (formerly `gen_ai.system`, deprecated) | — | ❌ Missing |
| `gen_ai.operation.name` | `gen_ai.agent.a2a.operation.name` | ⚠️ Prefixed |
| `gen_ai.conversation.id` | `gen_ai.agent.a2a.context_id` | ⚠️ Prefixed (code comment notes should use standard key) |
| `gen_ai.request.model` | — | ❌ Missing |
| `gen_ai.response.model` | — | ❌ Missing |
| `gen_ai.usage.input_tokens` | — | ❌ Missing |
| `gen_ai.usage.output_tokens` | — | ❌ Missing |
| `gen_ai.client.operation.duration` | — | ❌ Missing |
| `gen_ai.client.token.usage` | — | ❌ Missing |
| `server.address` / `server.port` | — | ❌ Missing |

Notably, the sample/example code uses the correct `gen_ai.*` keys for metric attributes, creating an inconsistency between the core module and the examples.

### Expected Behavior

- Use official `gen_ai.*` attribute keys as defined by the [OTel GenAI SemConv](https://opentelemetry.io/docs/specs/semconv/gen-ai/).
- For A2A-specific extensions that don't have an official SemConv equivalent, use a clearly namespaced prefix (e.g., `a2a.*`) rather than embedding under `gen_ai.agent.a2a.*`.
- Align the core module attributes with the example code (which already uses correct keys).

### Impact

- OTel GenAI-compatible dashboards (e.g., Grafana, Jaeger) cannot auto-discover and display A2A spans.
- Alerting rules built for `gen_ai.*` attributes do not match A2A traces.
- Users must build custom queries and dashboards instead of reusing existing GenAI observability tooling.
- Inconsistency between core module and example code creates confusion.

---

## Issue 7: SSE streaming operation duration is incomplete — span ends at stream initiation

### Summary

The current span for SSE streaming requests ends **immediately after the stream is initiated** (when the publisher returns), not when the stream actually completes. This means `gen_ai.client.operation.duration` (if it existed) would only record the stream setup time — potentially 0ms — rather than the full end-to-end streaming duration.

### Current Behavior

```
CLIENT Span: sendMessage

│ ← Span ends here (stream publisher created)

[CLIENT Span: event-1] (addLink)
[CLIENT Span: event-2] (addLink)
[CLIENT Span: event-3] (addLink)

│ ← Stream actually ends here (no span records this)

```

- The parent span closes before any events are sent.
- There is no span or metric that captures the full duration from stream initiation to the last event.
- The measured "operation duration" may be 0ms or near-0ms, completely misleading.

### Expected Behavior

- Either extend the parent span's lifetime to cover the full streaming duration, or
- Record a dedicated metric (`a2a.task.request.duration` or similar) that measures the time from stream initiation to the final event.

### Impact

- Operation latency appears as ~0ms in dashboards, masking actual processing time.
- Cannot measure true end-to-end streaming duration.
- SLO/SLA monitoring based on operation duration would be incorrect.

---

## Issue 8: No structured payload logging

### Summary

There is no structured logging of A2A protocol message payloads. The only option is an opt-in `toString()` on span attributes, which is not a proper logging mechanism.

### Current Behavior

- a2a-java has `gen_ai.agent.a2a.request` / `gen_ai.agent.a2a.response` span attributes (enabled via system property), but these are **span attributes, not structured logs**.
- There is no structured log event for task requests, state changes, result artifacts, negotiation payloads, authorization decisions, or notification deliveries.
- No payload size truncation or sensitive data masking is provided.

### Expected Behavior

- Provide structured log events (at DEBUG level by default) for key protocol messages: task requests, state changes, result artifacts, negotiation payloads, authorization decisions, notification subscriptions and deliveries.
- Include configurable payload size truncation (default 4KB) with `[truncated]` marker.
- Provide sensitive data masking with customizable strategies.
- Logs should be correlated to traces via `trace_id` / `span_id`.

### Impact

- Cannot inspect actual message payloads when debugging issues.
- No audit trail of authorization decisions or negotiation content.
- Cannot correlate log entries to traces for root-cause analysis.
- The `toString()` approach is not suitable for production use (no truncation, no masking, no structured fields).

---

## Summary Table

| # | Issue | Severity | Category |
| --- | --- | --- | --- |
| 1 | Server-side SSE per-event spans missing | High | Tracing Gap |
| 2 | Push notification delivery untraceable | High | Tracing Gap |
| 3 | No business-level semantic attributes | High | Semantic Gap |
| 4 | Task state transitions not tracked | Medium | Tracing Gap |
| 5 | No Metrics implementation in core module | High | Metrics Gap |
| 6 | Custom `gen_ai.agent.a2a.*` prefix vs official `gen_ai.*` SemConv | High | SemConv Misalignment |
| 7 | SSE streaming duration incomplete (span ends at initiation) | High | Tracing Gap |
| 8 | No structured payload logging | Medium | Logging Gap |

### Relevant log output

```shell

```

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.