open-telemetry / open-telemetry/opentelemetry-python-genai

google-genai: instrument_generate_content() unconditionally sets OTEL_INSTRUMENTATION_GENAI_EMIT_EVENT=true, overriding config and user setting

Open Beginner friendly
#619 8 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39
Forks
63
Avg merge
1d 15h
Merged PRs (30d)
175

Description

Package

opentelemetry-instrumentation-google-genai 1.0b0 (with opentelemetry-util-genai 1.0b0)

What happens

instrument_generate_content() unconditionally mutates a global process environment variable when instrumenting:

def instrument_generate_content(
    telemetry_handler: TelemetryHandler,
    generate_content_config_key_allowlist: AllowList,
) -> object:
    os.environ["OTEL_INSTRUMENTATION_GENAI_EMIT_EVENT"] = "true"
    ...

This has a few problems:

  1. It overrides the documented default behavior. opentelemetry.util.genai.utils.should_emit_event() is documented to default based on OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT:

    • NO_CONTENT or SPAN_ONLY → events default off
    • EVENT_ONLY or SPAN_AND_EVENT → events default on

    Because instrumenting force-sets OTEL_INSTRUMENTATION_GENAI_EMIT_EVENT=true (which should_emit_event() treats as highest priority), the gen_ai.client.inference.operation.details event is emitted for every request even in NO_CONTENT / SPAN_ONLY mode. In those modes the event carries no message content, so it just duplicates the span's attributes — redundant telemetry the user did not ask for.

  2. It overrides an explicit user setting. A user who sets OTEL_INSTRUMENTATION_GENAI_EMIT_EVENT=false to disable the event has their choice silently overwritten by instrument().

  3. It mutates global process state as a side effect of instrumenting. Writing to os.environ affects the whole process (and any other GenAI instrumentation reading the same variable), is order-dependent, and is not thread-safe.

Minimal reproduction
import os

os.environ["OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT"] = "SPAN_ONLY"
os.environ["OTEL_INSTRUMENTATION_GENAI_EMIT_EVENT"] = "false"  # explicitly disable the event

from opentelemetry.instrumentation.google_genai import GoogleGenAiSdkInstrumentor

GoogleGenAiSdkInstrumentor().instrument()

print(os.environ["OTEL_INSTRUMENTATION_GENAI_EMIT_EVENT"])
# -> "true"   (the user's "false" was overwritten)

After this, a generate_content call emits a gen_ai.client.inference.operation.details event even though the user asked for SPAN_ONLY capture and explicitly set EMIT_EVENT=false.

Expected

instrument() should not write to os.environ. Whether the event is emitted should be decided by should_emit_event() from the (unmodified) configuration:

  • default off for NO_CONTENT / SPAN_ONLY,
  • default on for EVENT_ONLY / SPAN_AND_EVENT,
  • and an explicit OTEL_INSTRUMENTATION_GENAI_EMIT_EVENT from the user always respected.

If there is a reason the event must be emitted regardless of capture mode, it would be clearer to pass that intent into the TelemetryHandler / invocation directly rather than mutating a global environment variable.

Context

Found while integrating this instrumentation into Pydantic Logfire. As a downstream workaround we snapshot OTEL_INSTRUMENTATION_GENAI_EMIT_EVENT before calling instrument() and restore it afterwards, but that only works because should_emit_event() reads the variable per request rather than at instrument time.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in instrumentation/opentelemetry-instrumentation-google-genai/src/opentelemetry/instrumentation/google_genai/generate_content.py at instrument_generate_content(), then trace how should_emit_event() reads configuration. Run the existing Google GenAI instrumentation tests and add coverage for explicit false and SPAN_ONLY or NO_CONTENT settings. Done means instrument() leaves os.environ unchanged and event emission follows the documented configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.