open-telemetry / open-telemetry/opentelemetry-ruby
Periodic Metric exporter unable to collect data points on Passenger-based servers
Nobody has claimed this yet.
- Dominant language
- Ruby
- Stars
- 606
- Forks
- 301
- Avg merge
- 3d 19h
- Merged PRs (30d)
- 42
Description
Description of the bug
Due to Passenger's smart spawning mechanism which creates worker processes by forking an initial pre-loader process, we notice that the thread for the periodic metric exporter is only attached to the pre-loader process and as such does not share memory with the other child processes which handle the actual incoming requests.
This causes the periodic metric reader to never have any data points since the metrics are stored in the memory of the child processes and are not shared with the thread which is attached to the pre-loader process.
The spawning mechanism, as well as its caveats, are described here: https://www.phusionpassenger.com/library/indepth/ruby/spawn_methods/#the-smart-spawning-method
To resolve this, the periodic metric reader needs to be initialised for each of the forked worker processes instantiated by Passenger, which was done as followed based on events that are made available within the internals of Passenger. This allowed us to send metrics as required.
if defined?(PhusionPassenger)
PhusionPassenger.on_event(:starting_worker_process) do |forked|
if forked
metric_reader = OpenTelemetry::SDK::Metrics::Export::PeriodicMetricReader.new(
exporter: OpenTelemetry::Exporter::OTLP::Metrics::MetricsExporter.new,
)
OpenTelemetry.meter_provider.add_metric_reader(metric_reader)
end
end
end
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Passenger starting_worker_process example in the issue and inspect the metric reader and exporter initialization paths in the SDK. Reproduce the behavior with Passenger smart spawning; done means each forked worker initializes a periodic metric reader and successfully exports data points.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ruby
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100