open-telemetry / open-telemetry/opentelemetry-python

Forking with PeriodicExportingMetricReader results in ValueError

Open
#3,402 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
2.6k
Forks
1k
Avg merge
4d 15h
Merged PRs (30d)
19

Description

Describe your environment
Python version: 3.10.4
grpc version: 1.43
Opentelemetry version

Steps to reproduce
Reproducer:

from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.metrics.view import ExplicitBucketHistogramAggregation, View
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.metrics import get_meter_provider
import functools
import threading

label = "main_test"
attrs = {SERVICE_NAME: "test.shiva"}
RESOURCE = Resource.create(attrs)
exporter = OTLPMetricExporter()
reader = PeriodicExportingMetricReader(exporter, export_interval_millis=10*1000, export_timeout_millis=5*1000)
provider = MeterProvider( metric_readers=[reader],  resource=RESOURCE, 
views=[ View( instrument_type=metrics.Histogram, instrument_name="*duration", 
aggregation=ExplicitBucketHistogramAggregation())])
metrics.set_meter_provider(provider)

meter = get_meter_provider().get_meter("test.shiva")
histogram = meter.create_histogram(label)
METRICS = {}
METRICS[label] = histogram


def record_data():
    from opentelemetry.metrics import get_meter_provider
    import os
    label = "main_test"
    histogram_metric = METRICS.get(label)
    histogram_metric.record(10, attributes={"app": str(os.getpid())})

record_data()

def child():
   record_data()
   print(f"Child: {list(threading.enumerate())}")
   while True:
       pass

def parent():
    newpid = os.fork()
    if newpid == 0:
        child()
    else:
        pids = (os.getpid(), newpid)
        print("parent: %d, child: %d\n" % pids)
        record_data()
        print(list(threading.enumerate()))

parent()

What is the expected behavior?
I'd expect the metrics of both child and parent process to be captured.

What is the actual behavior?
I see ValueError = ValueError: Cannot invoke RPC on closed channel!

Additional context
I'm setting couple of envs as suggested by grpc

GRPC_ENABLE_FORK_SUPPORT true
GRPC_POLL_STRATEGY epoll1

I notice the the Periodic Exporter spawns the threads on fork and child has different thread than the parent

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the supplied reproducer with PeriodicExportingMetricReader, OTLPMetricExporter, and os.fork(). Then inspect the metric reader and exporter fork/thread lifecycle to determine why the child invokes a closed gRPC channel. Done means both parent and child can record and export metrics without the ValueError, with regression coverage for the forked-process case.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, python
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.