[RFC]: End-to-End Transfer Tracing for Mooncake

### Changes proposed

## Summary

This RFC proposes a unified tracing architecture for Mooncake transfer execution.

The design introduces a reusable tracing subsystem and applies it across the transfer path so a single
logical transfer can be observed across:

- store-facing orchestration
- transfer-engine batch management
- multi-transport task submission
- per-slice data-plane execution
- terminal completion / failure / timeout reporting

The result is a structurally consistent trace tree that supports both live debugging and offline postmortem
analysis.

---

## Motivation

Mooncake transfer workflows cross several layers, but observability has historically been fragmented. Logs
and metrics can show symptoms, but they do not preserve causality.

This creates recurring problems:

- latency attribution is ambiguous
- control-plane and data-plane delays are hard to separate
- timeout root cause is difficult to localize
- store-level operations and transfer-engine execution are not naturally correlated
- external systems cannot reliably ingest or reconstruct transfer timelines

We need tracing that preserves parent-child relationships across the full transfer lifecycle, not just
isolated spans emitted by individual modules.

---

## Goals

- Provide a common tracing substrate for Mooncake components
- Preserve trace continuity across store, transfer engine, and transport layers
- Model transfer execution with stable, explicit span hierarchy
- Export traces to local files and OTLP collectors
- Support async export with retry and spool behavior
- Improve trace completeness for failure and timeout cases
- Make offline investigation possible from raw JSONL outputs

## Non-goals

- Redesign transfer scheduling or transport selection
- Replace existing metrics or logs
- Guarantee zero-overhead tracing on hot paths
- Fully redesign all observability ownership boundaries in one patch
- Solve every TENT-specific tracing gap in this RFC

---

## Background

A Mooncake transfer is not a single function call. It is a lifecycle:

1. a store-side operation decides how to move data
2. transfer-engine allocates and manages a batch
3. multi-transport selects concrete transports
4. transports submit slice-level work
5. status polling eventually determines terminal state

Without explicit trace propagation, each stage can emit useful local signals while the global story remains
invisible.

The core design principle of this RFC is:

**Make the lifecycle first-class.**
Do not infer it from logs after the fact.

---

## Proposal

### 1. Introduce `mooncake-tracing`

Add a dedicated tracing module that provides:

- `TraceContext`
- `TraceCarrier`
- `TraceRecord`
- `TracingFacade`
- RAII-style `Span`
- configurable sampling
- pluggable exporters

This module becomes the single place where trace creation, parenting, export, and sampling behavior live.

### 2. Define a stable transfer span hierarchy

The proposed hierarchy is:

- `mooncake.transfer.operation`
- `mooncake.transfer.submit_gap`
- `te.batch`
- `te.batch.submit`
- `te.task.submit`
- slice events
- `te.batch.execute`
- `te.task.status`
- `te.batch.status`
- `mooncake.transfer.wait_completion`
- `mooncake.transfer.complete`

This hierarchy captures both:
- the user-visible transfer operation lifecycle
- the internal execution lifecycle inside the transfer engine

### 3. Support parent context propagation

When an upstream trace context exists, Mooncake should attach to it.
When no parent exists, Mooncake should create its own operation root.

This allows Mooncake transfers to participate in larger distributed traces while still remaining self-
contained when invoked directly.

### 4. Keep batch-level tracing alive until terminal state

A short-lived submit-only span is insufficient because a transfer can spend most of its time after
submission.

The batch root span must survive until terminal completion, failure, cancellation, or timeout.
A separate persistent execution span (`te.batch.execute`) captures the data-plane lifetime.

### 5. Add explicit task and slice tracing state

`MultiTransport` maintains trace registry state for:

- batch context
- task context
- transport name
- slice queue state
- slice terminal state
- deduplicated terminal markers

This avoids duplicate terminal events and lets status polling produce complete traces even when terminal
evidence appears asynchronously.

### 6. Export traces through configurable backends

Supported paths include:

- in-memory exporter for tests
- JSONL exporter for local/debug workflows
- OTLP/HTTP exporter for collector integration
- async remote exporter with:
- queue limits
- retry
- local spool fallback

### 7. Provide offline trace aggregation

An offline aggregator script merges JSONL outputs into a readable waterfall / phase report for postmortem
analysis.

This is especially useful when traces were produced locally or remote collection was unavailable.

---

## Detailed Design

### Store-side tracing

Store transfer flows create a `TransferTraceSession` that:

- creates a standalone root span if no parent exists
- reuses the provided parent context when one exists
- tracks submit-gap timing
- creates a wait-completion span
- records final completion status

This ensures the store layer has a coherent operation-level narrative.

### Transfer-engine tracing

`TransferEngineImpl` owns batch-lifecycle tracing semantics.

It maintains active batch trace state and ensures:

- one stable `te.batch` root per batch
- one stable `te.batch.execute` span for data-plane lifetime
- child operation spans for submit / notify / status actions
- terminal closure at batch completion or release

### Multi-transport tracing

`MultiTransport` captures execution details that are invisible at higher layers:

- transport selection per task
- queued slices
- task terminal status
- batch terminal status
- timeout-specific terminal evidence

It also improves completeness by observing slice state directly, not just trusting aggregated counters.

### Sampling semantics

Structural spans are tagged with explicit sampling priority so that the trace skeleton is preserved more
reliably.

The intent is:

- keep the lifecycle readable
- avoid partial traces where the root exists but critical structural spans do not

### Export and durability semantics

The exporter path is designed so that tracing failure does not become transfer failure.

Key properties:

- tracing is config-gated
- remote export is async
- retries are bounded
- fallback to local persistence is available
- offline reconstruction remains possible from JSONL output

---

## Configuration

Relevant configuration knobs include:

- `MC_TRACING_ENABLED`
- `MC_TRACING_EXPORTER`
- `MC_TRACING_JSONL_PATH`
- `MC_TRACING_REMOTE_ENDPOINTS`
- `MC_TRACING_EXPORTER_QUEUE_MAX_ITEMS`
- `MC_TRACING_EXPORTER_QUEUE_MAX_BYTES`
- `MC_TRACING_EXPORTER_RETRY_BASE_MS`
- `MC_TRACING_EXPORTER_RETRY_MAX_MS`
- `MC_TRACING_EXPORTER_RETRY_MAX_ATTEMPTS`
- `MC_TRACING_EXPORTER_SPOOL_DIR`
- `MC_TRACING_SAMPLING_MODE`
- `MC_TRACING_SAMPLING_BASE_RATIO`
- `MC_TRACING_SLOW_THRESHOLD_MS`
- `OTEL_EXPORTER_OTLP_ENDPOINT`
- `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`
- `OTEL_EXPORTER_OTLP_HEADERS`
- `OTEL_EXPORTER_OTLP_TRACES_HEADERS`
- `OTEL_EXPORTER_OTLP_TIMEOUT`
- `OTEL_EXPORTER_OTLP_TRACES_TIMEOUT`

---

## Expected Benefits

- Faster debugging of latency and timeout issues
- Better cross-layer causality
- Easier collector integration
- More reliable postmortem reconstruction
- Stronger testability of observability semantics

---

## Trade-offs

### Pros

- Much better visibility into transfer execution
- Clear causal relationships across layers
- Works for both online and offline analysis
- Testable tracing lifecycle semantics

### Cons

- More code in hot transfer paths
- More lifecycle state to coordinate
- More observability concepts for contributors to understand
- Some ownership remains split across store and transfer engine

---

## Alternatives Considered

### 1. Logs only
Rejected because logs are not hierarchical and do not preserve parent-child causality well.

### 2. Metrics only
Rejected because metrics are good for aggregates, not per-transfer diagnosis.

### 3. Trace only the store layer
Rejected because most execution detail lives below store.

### 4. Trace only the transport layer
Rejected because transport-only traces lose user-visible operation context.

### 5. Emit short-lived submit spans only
Rejected because this hides the long-running part of transfer execution.

---

## Risks and Mitigations

### Risk: tracing overhead
Mitigation:
- config-gated enablement
- async exporters
- bounded queues

### Risk: incomplete traces under collector failure
Mitigation:
- JSONL fallback
- spool support
- offline aggregation tooling

### Risk: duplicate terminal events
Mitigation:
- explicit registry-based terminal deduplication

### Risk: lifecycle ownership remains split
Mitigation:
- stable hierarchy now
- follow-up refactor later if needed

---

## Rollout Plan

1. Land tracing module and transfer-path instrumentation
2. Enable tracing in development and targeted test environments
3. Validate JSONL and OTLP outputs with real transfer flows
4. Tune sampling and exporter defaults
5. Expand integration coverage
6. Consider broader production rollout after overhead review

---

## Testing Strategy

This work should be validated through:

- unit tests for context propagation
- unit tests for registry deduplication
- transfer-task tests for store-side tracing behavior
- transport tests for timeout / terminal semantics
- manual JSONL inspection
- collector ingestion tests when OTLP is enabled

---

## Open Questions

- Should TENT mode fully preserve external parent contexts now or in a follow-up?
- Do we want a stable trace schema document for downstream tooling?
- Should structural spans always bypass ratio sampling?
- Should the batch/task/slice trace registries eventually move into a single shared lifecycle component?

---

## Future Work

- end-to-end trace assertions in integration tests
- trace schema documentation
- dashboards / canned queries for common transfer failure modes
- deeper collector-side correlation with metrics and logs
- further simplification of cross-layer trace ownership

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)