grafana / grafana/otel-profiling-ruby
Adding Pyroscope::Otel::SpanProcessor disables all OpenTelemetry trace exporters and Rails OTel logging
- Dominant language
- Ruby
- Stars
- 9
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
When I add the following line to my OpenTelemetry SDK configuration:
```ruby
config.add_span_processor(
Pyroscope::Otel::SpanProcessor.new("#{APP_NAME}.cpu", PYROSCOPE_ENDPOINT)
)
```
OpenTelemetry stops exporting traces completely.
The Rails application starts successfully, there are no exceptions, and requests are processed normally, but no traces are exported (neither OTLP nor the console exporter).
Removing this line immediately restores trace exporting.
In Grafana UI `Loki` shows `logs` but there is not `traces` in `Tempo`.
## Environment
* Ruby: 3.4.2
* Rails: 8.0.2
* `opentelemetry-sdk`: 1.12.0
* `pyroscope`: 1.0.9
* `pyroscope-otel`: 0.2.0
## Expected behaviour
`Pyroscope::Otel::SpanProcessor` should be added alongside the existing OpenTelemetry span processors, allowing traces to continue being exported while also enabling profile correlation.
## Actual behaviour
After adding `Pyroscope::Otel::SpanProcessor`, trace exporting stops entirely without any error messages.
P.S.
My Rails repo is based on the well-known Rails OpenTelemetry Workshop. I'm just experimenting and trying to extend it with the Grafana stack.
* [Repo: 2025-railsconf-opentelemetry-workshop](https://github.com/kaylareopelle/2025-railsconf-opentelemetry-workshop)
* [Video](https://www.youtube.com/watch?v=E_rKGBN_caQ)
---
### Attachments:
- [before_with_otel_rails_logs.txt](https://github.com/user-attachments/files/29646913/before_with_otel_rails_logs.txt)
- [after_without_otel_rails_logs.txt](https://github.com/user-attachments/files/29646919/after_without_otel_rails_logs.txt)
**Gemfile**:
```rb
# Traces
gem "opentelemetry-sdk", "~> 1.12"
gem "opentelemetry-exporter-otlp", "~> 0.34.0"
gem "opentelemetry-instrumentation-all", "~> 0.94.0"
# Metrics
gem "opentelemetry-metrics-sdk"
gem "opentelemetry-exporter-otlp-metrics"
# Logs
gem "opentelemetry-logs-sdk"
gem "opentelemetry-exporter-otlp-logs"
gem "opentelemetry-instrumentation-logger"
# Profiling
gem 'pyroscope' # 1.0.9
gem 'pyroscope-otel' # 0.2.0
```
**config/initializers/opentelemetry.rb**:
```rb
require 'pyroscope'
APP_NAME = "hike-tracker"
# PYROSCOPE_ENDPOINT = "http://pyroscope:4040" # Docker env
PYROSCOPE_ENDPOINT = "http://localhost:4040" # Dev env
# Exporting to New Relic
ENV['NEW_RELIC_LICENSE_KEY'] = 'eu01x03...'
ENV['OTEL_SERVICE_NAME'] = 'test-rails-app'
ENV['OTEL_RESOURCE_ATTRIBUTES'] = 'service.instance.id=123'
ENV["OTEL_EXPORTER_OTLP_ENDPOINT"] = "http://localhost:4318" # to export to collector & plain file
# ENV['OTEL_EXPORTER_OTLP_ENDPOINT'] = 'https://otlp.eu01.nr-data.net'
# ENV['OTEL_EXPORTER_OTLP_HEADERS'] = "api-key=#{ENV['NEW_RELIC_LICENSE_KEY']}"
ENV['OTEL_EXPORTER_OTLP_COMPRESSION'] = 'gzip'
ENV['OTEL_EXPORTER_OTLP_PROTOCOL'] = 'http/protobuf'
ENV['OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT'] = '4095'
# Metrics
ENV["OTEL_METRIC_EXPORT_INTERVAL"] = "3000" # 3 sec * 100 ms
# Add console exporter
ENV['OTEL_TRACES_EXPORTER'] = 'console,otlp'
# Logs
ENV["OTEL_LOGS_EXPORTER"] = "otlp"
Pyroscope.configure do |config|
config.application_name = APP_NAME
config.server_address = PYROSCOPE_ENDPOINT
config.autoinstrument_rails = true
end
# Configure the SDK
OpenTelemetry::SDK.configure do |config|
config.service_name = APP_NAME
config.service_version = "1.0.0"
# ---------------- BUG: this line breaks Open Telemetry logic
# config.add_span_processor(Pyroscope::Otel::SpanProcessor.new("#{APP_NAME}.cpu", PYROSCOPE_ENDPOINT))
# ---------------
# Installs instrumentation for all available libraries
config.use_all({
"OpenTelemetry::Instrumentation::Rack" => {
allowed_response_headers: [ "Content-Type" ]
}
})
end
# Create a tracer specific to this application to create spans/traces
APP_TRACER = OpenTelemetry.tracer_provider.tracer(APP_NAME)
# Create a meter for this application to record metrics
meter = OpenTelemetry.meter_provider.meter(APP_NAME)
# Save the counter as a constant to access it outside the initializer
HIKE_COUNTER = meter.create_counter("activities.completed", unit: "activity", description: "Number of activities completed")
# Create a histogram with a view
explicit_boundaries = [ 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10 ]
OpenTelemetry.meter_provider.add_view("http.server.request.duration",
type: :histogram,
aggregation: OpenTelemetry::SDK::Metrics::Aggregation::ExplicitBucketHistogram.new(
boundaries: explicit_boundaries
)
)
duration_histogram = meter.create_histogram("http.server.request.duration", unit: "ms", description: "Duration of HTTP server requests.")
# Subscribe to an ActiveSupport notification to add a metric defined by Semantic Conventions that's not recorded by instrumentation yet
ActiveSupport::Notifications.subscribe "process_action.action_controller" do |event|
payload = event.payload
server_protocol = payload[:headers]["SERVER_PROTOCOL"].split("/")
attributes = {
"controller.action" => "#{payload[:controller]}##{payload[:action]}",
"http.request.method" => payload[:method],
# 'url.scheme' => ,
# 'error.type' => ,
"http.response.status.code" => payload[:status],
"http.route" => payload[:path],
"network.protocol.name" => server_protocol[0],
"network.protocol.version" => server_protocol[1],
"server.address" => payload[:headers]["Host"],
"server.port" => payload[:headers]["SERVER_PORT"]
}
duration_histogram.record(event.duration, attributes: attributes)
end
```
**docker-compose.yml**:
```yml
services:
grafana: # Telemetry UI
image: grafana/grafana:latest # 13.1
container_name: grafana
ports:
- "3333:3000" # Admin UI
depends_on:
- tempo
loki: # Logs
image: grafana/loki:latest # 3.7.3
container_name: loki
ports:
- "3100:3100"
command:
- "-config.file=/etc/loki/local-config.yaml"
otel-collector: # OpenTelemetry Hub
image: otel/opentelemetry-collector-contrib:latest
container_name: otel-collector
command:
- "--config=/etc/otelcol-contrib/config.yaml"
depends_on:
- tempo
volumes:
- ./config/otel-collector.yml:/etc/otelcol-contrib/config.yaml
- ./otel-data:/otel-data
ports:
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
prometheus: # Metrics
image: prom/prometheus:latest # 3.13
container_name: prometheus
ports:
- "9090:9090"
volumes:
- ./config/otel-prometheus.yml:/etc/prometheus/prometheus.yml:ro
command:
- --config.file=/etc/prometheus/prometheus.yml
depends_on:
- otel-collector
pyroscope:
image: grafana/pyroscope:latest # 2.1
container_name: pyroscope
ports:
- "4040:4040"
tempo: # Traces
image: grafana/tempo:latest # 2.7.2
container_name: tempo
command:
- "-config.file=/etc/tempo.yaml"
ports:
- "3200:3200"
- "4319:4317" # OTLP gRPC
volumes:
- ./config/otel-tempo.yml:/etc/tempo.yaml
- ./tempo-data:/var/tempo
```
**Gemfile.lock**:
```rb
...
opentelemetry-api (1.10.0)
logger
opentelemetry-common (0.25.0)
opentelemetry-api (~> 1.0)
opentelemetry-exporter-otlp (0.34.0)
google-protobuf (>= 3.18)
googleapis-common-protos-types (~> 1.3)
opentelemetry-api (~> 1.1)
opentelemetry-common (~> 0.20)
opentelemetry-sdk (~> 1.10)
opentelemetry-semantic_conventions
opentelemetry-exporter-otlp-logs (0.5.1)
google-protobuf (>= 3.18)
googleapis-common-protos-types (~> 1.3)
opentelemetry-api (~> 1.1)
opentelemetry-common (~> 0.20)
opentelemetry-logs-api (~> 0.1)
opentelemetry-logs-sdk (~> 0.1)
opentelemetry-sdk
opentelemetry-semantic_conventions
opentelemetry-exporter-otlp-metrics (0.10.0)
google-protobuf (>= 3.18, < 5.0)
googleapis-common-protos-types (~> 1.3)
opentelemetry-api (~> 1.1)
opentelemetry-common (~> 0.20)
opentelemetry-metrics-api (~> 0.2)
opentelemetry-metrics-sdk (~> 0.5)
opentelemetry-sdk (~> 1.2)
opentelemetry-semantic_conventions
opentelemetry-helpers-mysql (0.6.0)
opentelemetry-api (~> 1.7)
opentelemetry-common (~> 0.21)
opentelemetry-helpers-sql (0.4.0)
opentelemetry-api (~> 1.7)
opentelemetry-helpers-sql-processor (0.5.0)
opentelemetry-api (~> 1.0)
opentelemetry-common (~> 0.21)
opentelemetry-instrumentation-action_mailer (0.8.1)
opentelemetry-instrumentation-active_support (~> 0.10)
opentelemetry-instrumentation-action_pack (0.18.0)
opentelemetry-instrumentation-rack (~> 0.29)
opentelemetry-instrumentation-action_view (0.13.0)
opentelemetry-instrumentation-active_support (~> 0.10)
opentelemetry-instrumentation-active_job (0.13.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-active_model_serializers (0.25.0)
opentelemetry-instrumentation-active_support (>= 0.7.0)
opentelemetry-instrumentation-active_record (0.13.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-active_storage (0.5.1)
opentelemetry-instrumentation-active_support (~> 0.10)
opentelemetry-instrumentation-active_support (0.12.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-all (0.94.0)
opentelemetry-instrumentation-active_model_serializers (~> 0.25.0)
opentelemetry-instrumentation-anthropic (~> 0.5.0)
opentelemetry-instrumentation-aws_lambda (~> 0.7.0)
opentelemetry-instrumentation-aws_sdk (~> 0.12.0)
opentelemetry-instrumentation-bunny (~> 0.25.0)
opentelemetry-instrumentation-concurrent_ruby (~> 0.25.0)
opentelemetry-instrumentation-dalli (~> 0.30.0)
opentelemetry-instrumentation-delayed_job (~> 0.26.0)
opentelemetry-instrumentation-ethon (~> 0.29.0)
opentelemetry-instrumentation-excon (~> 0.29.0)
opentelemetry-instrumentation-faraday (~> 0.33.0)
opentelemetry-instrumentation-grape (~> 0.7.0)
opentelemetry-instrumentation-graphql (~> 0.32.0)
opentelemetry-instrumentation-grpc (~> 0.5.0)
opentelemetry-instrumentation-gruf (~> 0.6.0)
opentelemetry-instrumentation-http (~> 0.30.0)
opentelemetry-instrumentation-http_client (~> 0.29.0)
opentelemetry-instrumentation-httpx (~> 0.8.0)
opentelemetry-instrumentation-koala (~> 0.24.0)
opentelemetry-instrumentation-lmdb (~> 0.26.0)
opentelemetry-instrumentation-mongo (~> 0.26.0)
opentelemetry-instrumentation-mysql2 (~> 0.34.0)
opentelemetry-instrumentation-net_http (~> 0.29.0)
opentelemetry-instrumentation-pg (~> 0.36.0)
opentelemetry-instrumentation-que (~> 0.13.0)
opentelemetry-instrumentation-racecar (~> 0.7.0)
opentelemetry-instrumentation-rack (~> 0.31.0)
opentelemetry-instrumentation-rails (~> 0.42.0)
opentelemetry-instrumentation-rake (~> 0.6.0)
opentelemetry-instrumentation-rdkafka (~> 0.10.0)
opentelemetry-instrumentation-redis (~> 0.29.0)
opentelemetry-instrumentation-resque (~> 0.9.0)
opentelemetry-instrumentation-restclient (~> 0.28.0)
opentelemetry-instrumentation-ruby_kafka (~> 0.25.0)
opentelemetry-instrumentation-sidekiq (~> 0.29.0)
opentelemetry-instrumentation-sinatra (~> 0.30.0)
opentelemetry-instrumentation-trilogy (~> 0.69.0)
opentelemetry-instrumentation-anthropic (0.5.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-aws_lambda (0.7.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-aws_sdk (0.12.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-base (0.26.1)
opentelemetry-api (~> 1.7)
opentelemetry-common (~> 0.21)
opentelemetry-registry (~> 0.1)
opentelemetry-instrumentation-bunny (0.25.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-concurrent_ruby (0.25.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-dalli (0.30.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-delayed_job (0.26.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-ethon (0.29.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-excon (0.29.1)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-faraday (0.33.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-grape (0.7.0)
opentelemetry-instrumentation-rack (~> 0.29)
opentelemetry-instrumentation-graphql (0.32.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-grpc (0.5.1)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-gruf (0.6.1)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-http (0.30.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-http_client (0.29.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-httpx (0.8.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-koala (0.24.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-lmdb (0.26.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-logger (0.4.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-logs-api (~> 0.1)
opentelemetry-instrumentation-mongo (0.26.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-mysql2 (0.34.0)
opentelemetry-helpers-mysql
opentelemetry-helpers-sql
opentelemetry-helpers-sql-processor
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-net_http (0.29.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-pg (0.36.0)
opentelemetry-helpers-sql
opentelemetry-helpers-sql-processor
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-que (0.13.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-racecar (0.7.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-rack (0.31.1)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-rails (0.42.0)
opentelemetry-instrumentation-action_mailer (~> 0.7)
opentelemetry-instrumentation-action_pack (~> 0.17)
opentelemetry-instrumentation-action_view (~> 0.12)
opentelemetry-instrumentation-active_job (~> 0.11)
opentelemetry-instrumentation-active_record (~> 0.12)
opentelemetry-instrumentation-active_storage (~> 0.4)
opentelemetry-instrumentation-active_support (~> 0.11)
opentelemetry-instrumentation-concurrent_ruby (~> 0.25)
opentelemetry-instrumentation-rake (0.6.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-rdkafka (0.10.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-redis (0.29.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-resque (0.9.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-restclient (0.28.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-ruby_kafka (0.25.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-sidekiq (0.29.0)
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-instrumentation-sinatra (0.30.0)
opentelemetry-instrumentation-rack (~> 0.29)
opentelemetry-instrumentation-trilogy (0.69.0)
opentelemetry-helpers-mysql
opentelemetry-helpers-sql
opentelemetry-helpers-sql-processor
opentelemetry-instrumentation-base (~> 0.25)
opentelemetry-semantic_conventions (>= 1.8.0)
opentelemetry-logs-api (0.4.0)
opentelemetry-api (~> 1.0)
opentelemetry-logs-sdk (0.6.0)
opentelemetry-api (~> 1.2)
opentelemetry-logs-api (~> 0.1)
opentelemetry-sdk (~> 1.3)
opentelemetry-metrics-api (0.6.0)
opentelemetry-api (~> 1.0)
opentelemetry-metrics-sdk (0.15.0)
opentelemetry-api (~> 1.1)
opentelemetry-metrics-api (~> 0.2)
opentelemetry-sdk (~> 1.2)
opentelemetry-registry (0.6.0)
opentelemetry-api (~> 1.1)
opentelemetry-sdk (1.12.0)
logger
opentelemetry-api (~> 1.1)
opentelemetry-common (~> 0.20)
opentelemetry-registry (~> 0.2)
opentelemetry-semantic_conventions
opentelemetry-semantic_conventions (1.41.0)
opentelemetry-api (~> 1.0)
...
pyroscope (1.0.9)
ffi
pyroscope (1.0.9-aarch64-linux)
ffi
pyroscope (1.0.9-arm64-darwin)
ffi
pyroscope (1.0.9-x86_64-darwin)
ffi
pyroscope (1.0.9-x86_64-linux)
ffi
pyroscope-otel (0.2.0)
opentelemetry-api (~> 1.1)
pyroscope (>= 1.0, < 2.0)
...
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the issue using the provided config/initializers/opentelemetry.rb, Gemfile, and Gemfile.lock versions, with and without Pyroscope::Otel::SpanProcessor. Start by locating the SpanProcessor implementation and its tests in this repository, then verify that OTLP and console trace exporters still receive spans when it is registered alongside existing processors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ruby
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100