[Bug]: We observed inconsistent OpenTelemetry metric exports in Salt master deployments using multiple MWorkers.

Summary

We observed inconsistent values for the Grafana dashboards tracking:

• Total Jobs Published
• Total Jobs Completed

The counters were not increasing consistently and would fluctuate depending on which Salt master worker process handled the event.

After investigation, we found that the issue was related to OpenTelemetry metrics being initialized before worker processes were forked.

Problem

In a multi-worker Salt master deployment:

• Job events are processed by multiple MWorkers.
• OpenTelemetry metrics state is inherited from the master process.
• Worker processes reuse metrics provider/exporter state created by the parent.
• Each worker maintains its own process-local metric state.
• Metrics generated by workers are not exported consistently.

As a result:

• Grafana dashboards showed fluctuating values.
• salt.jobs.published was not reported consistently.
• salt.jobs.completed was not reported consistently.
• Counters appeared incomplete depending on which worker handled the job.
• Dashboard totals were unreliable and non-monotonic.

Expected Behavior

Metrics should provide an accurate cluster-wide view of job activity.

For example:

• Every published job should increment salt.jobs.published exactly once.
• Every completed job return should increment salt.jobs.completed exactly once.
• Counter values should increase monotonically.
• Grafana dashboards should display consistent totals regardless of which worker processed the event.

Each worker should maintain its own properly initialized OpenTelemetry provider/exporter state.

Actual Behavior

OpenTelemetry components were initialized before worker processes were forked.

Worker processes inherited the parent process state and continued operating with that inherited state.

This resulted in:

• Inconsistent metric exports.
• Fragmented counter values across workers.
• Missing increments for some job events.
• Fluctuating Grafana dashboard values.
• Unreliable totals for Jobs Published and Jobs Completed.

Investigation

We traced the issue to OpenTelemetry initialization in a forked multi-process environment.

Workers inherited MeterProvider and exporter state from the master process.

Additionally, job counters were being generated in worker processes, which meant metric values became dependent on worker-specific state rather than being tracked from a single authoritative source.

Resolution

We resolved the issue by:

1. Reinitializing OpenTelemetry provider/exporter state independently in each worker process after a fork.
2. Moving job counters to the EventMonitor process, which observes all job events exactly once.

After applying these changes:

• salt.jobs.published became stable and monotonic.
• salt.jobs.completed became stable and monotonic.
• Grafana dashboards no longer fluctuated.
• Metrics were exported consistently across workers.
• No exporter-state conflicts were observed.

Questions

1. Is this behavior already known or documented?
2. Has a similar issue been reported previously?
3. Is worker-level OpenTelemetry initialization the recommended approach for Salt masters running multiple MWorkers?
4. Would documentation around OpenTelemetry and process forking be beneficial?

The issue names OpenTelemetry initialization, MeterProvider/exporter state, MWorkers, and the EventMonitor, but no files or tests. Start by tracing those components through Salt's multi-worker path and verify how job events update the published and completed counters. Done means worker metrics are independently initialized and the Grafana totals remain complete and monotonic.