demml / demml/scouter

Support stable Scouter tracing provider reconfiguration

Open
#279 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
13
Forks
1
PR merge metrics
No merged PRs in 30d

Description

# Stable Scouter Tracing Provider

## Problem

Scouter's current tracing lifecycle assumes the global OpenTelemetry provider is installed, used, and then torn down. That works for normal production applications that call `ScouterInstrumentor().instrument(...)` once at process startup.

It breaks down in long-lived Python processes that reconfigure tracing after frameworks have already cached tracers:

- test suites that repeatedly instrument and uninstrument in one process
- notebooks that reconfigure tracing across cells
- plugin systems that load and unload tracing setup
- applications that need to change exporter or transport configuration without restarting

The ADK CI failure exposed this. Google ADK cached a module-level tracer, one test shut down Scouter's provider, a later test created a new provider, and ADK continued using the tracer bound to the old backend. The agent ran, but ADK spans never reached the new GenAI span backend.

## Design Goal

Scouter tracers should behave like stable handles. Frameworks should be able to cache a tracer once and keep using it after Scouter reconfiguration.

The design should stay framework agnostic. Scouter should not patch ADK, LangChain, FastAPI, or any other package-specific module globals in production code.

## Proposed Design

Install one stable Python `ScouterTracerProvider` as the process-global OpenTelemetry provider. Do not replace that global provider object on reconfiguration.

The provider owns replaceable backend state:

```text
OpenTelemetry global provider
-> Stable ScouterTracerProvider
-> current Rust SdkTracerProvider / exporter / transport
-> generation counter
```

Each `ScouterTracer` stores its instrumentation scope and lazily refreshes its Rust `BaseTracer` when the provider generation changes:

```text
ScouterTracer
scope = (name, version, schema_url, attributes)
cached_generation = 1
cached_base_tracer = BaseTracer for generation 1

start_span(...)
if provider.generation != cached_generation:
cached_base_tracer = provider.build_base_tracer(scope)
cached_generation = provider.generation

cached_base_tracer.start_span(...)
```

This means a framework-cached tracer remains valid. After Scouter reconfigures, the next span refreshes the tracer's underlying Rust backend without framework-specific cleanup.

## Rust Implementation Sketch

In `crates/scouter_tracing/src/tracer.rs`:

- Replace the current "provider already exists, no-op" behavior in `configure_tracing(...)`.
- Build a new `SdkTracerProvider` from the requested resource/exporter/transport config.
- Atomically swap it into `TRACER_PROVIDER_STORE`.
- Increment an atomic generation counter.
- Flush and shut down the old provider outside the write lock.
- Expose a PyO3 function for the current generation.

Sketch:

```rust
static TRACER_PROVIDER_STORE: RwLock>> = RwLock::new(None);
static TRACER_GENERATION: AtomicU64 = AtomicU64::new(0);

#[pyfunction]
pub fn configure_tracing(...) -> Result {
let new_provider = build_provider(...)?;

let old_provider = {
let mut guard = TRACER_PROVIDER_STORE
.write()
.map_err(|e| TraceError::PoisonError(e.to_string()))?;
guard.replace(Arc::new(new_provider))
};

let generation = TRACER_GENERATION.fetch_add(1, Ordering::SeqCst) + 1;

if let Some(provider) = old_provider {
provider.force_flush()?;
if let Err(err) = provider.shutdown() {
tracing::warn!("Failed to shut down replaced tracer provider: {err}");
}
}

Ok(generation)
}

#[pyfunction]
pub fn tracer_generation() -> u64 {
TRACER_GENERATION.load(Ordering::SeqCst)
}
```

`BaseTracer::new(...)` can continue to construct an `SdkTracer` from the currently stored provider. The important change is that the currently stored provider can change over time.

## Python Implementation Sketch

In `py-scouter/python/scouter/tracing/__init__.py`:

- Make `ScouterTracerProvider` the stable shell.
- Let `ScouterTracerProvider.configure(...)` call Rust `configure_tracing(...)` and update provider generation.
- Cache `ScouterTracer` by instrumentation scope.
- Make `ScouterTracer` store its scope and refresh the underlying Rust `BaseTracer` when generation changes.
- Make `ScouterInstrumentor.instrument(...)` install the global provider once. Later calls should reconfigure the existing Scouter provider rather than resetting OpenTelemetry internals and replacing the global provider object.

Sketch:

```python
class ScouterTracer:
def __init__(self, provider, name, version=None, schema_url=None, attributes=None):
self._provider = provider
self._scope = (name, version, schema_url, attributes)
self._generation = -1
self._base = None
self._lock = threading.Lock()

def _refresh_if_needed(self) -> None:
generation = self._provider.generation
if self._generation == generation:
return

with self._lock:
generation = self._provider.generation
if self._generation != generation:
self._base = self._provider.build_base_tracer(*self._scope)
self._generation = generation

def start_span(self, *args, **kwargs):
self._refresh_if_needed()
return self._base.start_span(*args, **kwargs)
```

## API Direction

Keep `instrument(...)` backward compatible, but make the lifecycle explicit:

```python
instrumentor = ScouterInstrumentor()
instrumentor.instrument(...) # installs stable provider if needed
instrumentor.reconfigure(...) # swaps backend/exporter/transport
instrumentor.uninstrument() # flushes/shuts down current backend and resets Scouter state
```

`instrument(...)` can call `reconfigure(...)` when a Scouter provider is already installed.

## Why This Is Better

- Framework agnostic: no ADK-specific production patching.
- Compatible with libraries that cache tracers.
- Better aligned with OpenTelemetry's process-global provider model.
- Supports repeated reconfiguration in tests, notebooks, plugin systems, and long-lived applications.
- Avoids repeatedly poking private OpenTelemetry internals to replace the global provider.

## Risks

- Shutdown and reconfiguration must avoid holding locks while flushing or shutting down exporters.
- In-flight spans may start on one generation and end after another generation is installed. This needs explicit test coverage.
- Existing tests may assume provider replacement semantics and need updates.
- PyO3 APIs and stubs need to be updated if new public functions are exposed.

## Verification Plan

- Add a test where a tracer is cached before reconfiguration, Scouter is reconfigured, and the cached tracer writes to the new backend.
- Add an ADK regression test that does not patch ADK module globals.
- Run Rust tracing crate tests.
- Rebuild the Python extension after Rust changes.
- Run Python tracing integration tests.
- Run full lint sequence required by the repository.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.