open-telemetry / open-telemetry/opentelemetry-python
Forking with PeriodicExportingMetricReader results in ValueError
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 1k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 19
Description
Describe your environment
Python version: 3.10.4
grpc version: 1.43
Opentelemetry version
Steps to reproduce
Reproducer:
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.metrics.view import ExplicitBucketHistogramAggregation, View
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.metrics import get_meter_provider
import functools
import threading
label = "main_test"
attrs = {SERVICE_NAME: "test.shiva"}
RESOURCE = Resource.create(attrs)
exporter = OTLPMetricExporter()
reader = PeriodicExportingMetricReader(exporter, export_interval_millis=10*1000, export_timeout_millis=5*1000)
provider = MeterProvider( metric_readers=[reader], resource=RESOURCE,
views=[ View( instrument_type=metrics.Histogram, instrument_name="*duration",
aggregation=ExplicitBucketHistogramAggregation())])
metrics.set_meter_provider(provider)
meter = get_meter_provider().get_meter("test.shiva")
histogram = meter.create_histogram(label)
METRICS = {}
METRICS[label] = histogram
def record_data():
from opentelemetry.metrics import get_meter_provider
import os
label = "main_test"
histogram_metric = METRICS.get(label)
histogram_metric.record(10, attributes={"app": str(os.getpid())})
record_data()
def child():
record_data()
print(f"Child: {list(threading.enumerate())}")
while True:
pass
def parent():
newpid = os.fork()
if newpid == 0:
child()
else:
pids = (os.getpid(), newpid)
print("parent: %d, child: %d\n" % pids)
record_data()
print(list(threading.enumerate()))
parent()
What is the expected behavior?
I'd expect the metrics of both child and parent process to be captured.
What is the actual behavior?
I see ValueError = ValueError: Cannot invoke RPC on closed channel!
Additional context
I'm setting couple of envs as suggested by grpc
GRPC_ENABLE_FORK_SUPPORT true
GRPC_POLL_STRATEGY epoll1
I notice the the Periodic Exporter spawns the threads on fork and child has different thread than the parent
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the supplied reproducer with PeriodicExportingMetricReader, OTLPMetricExporter, and os.fork(). Then inspect the metric reader and exporter fork/thread lifecycle to determine why the child invokes a closed gRPC channel. Done means both parent and child can record and export metrics without the ValueError, with regression coverage for the forked-process case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, python
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100