open-telemetry / open-telemetry/opentelemetry-java-instrumentation

IBM MQ: report the queue manager identifier and messaging.system on JMS spans

Open
#19,979 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement needs triage
Dominant language
Java
Stars
2.6k
Forks
1.2k
Avg merge
2d 18h
Merged PRs (30d)
228

Description

Is your feature request related to a problem? Please describe.

PR: https://github.com/open-telemetry/opentelemetry-java-instrumentation/pull/19896
Two hosts can each run an IBM MQ queue manager named QM1. In a federated MQ estate that is normal rather than an edge case. A span from the generic JMS instrumentation carries the destination and at best a queue manager name, so I cannot tell which of those two an application published to, and I cannot join that span to queue manager level infrastructure telemetry, because the only shared key is a name that collides.

Those spans also report messaging.system as jms, which describes the API the application used rather than the broker behind it. A consumer of the telemetry cannot tell IBM MQ traffic apart from ActiveMQ or any other JMS provider, and cannot correlate it with IBM MQ instrumentation in another language.

IBM MQ already has a globally unique identifier for the first problem: MQCA_Q_MGR_IDENTIFIER, selector 2032, formatted <QUEUE_MANAGER_NAME>_YYYY-MM-DD_HH.MM.SS, minted when the queue manager is created and distinct across hosts. It has no equivalent in the JMS API, so the generic JMS instrumentation cannot supply it and should not have to.

Describe the solution you'd like

A new javaagent instrumentation module, ibmmq, that enriches the spans the JMS instrumentation already creates. It never creates, ends, or otherwise alters a span, it only sets attributes. Everything below sits behind otel.instrumentation.ibmmq.experimental-span-attributes and is off by default.

It adds messaging.ibmmq.queue_manager.id, a string. The IBM MQ client caches the queue manager identity on the connection at MQCONN time and exposes it through its own property bag, JmsReadablePropertyContext.getStringProperty with CommonConstants.WMQ_RESOLVED_QUEUE_MANAGER_ID for the javax client and the same property under com.ibm.msg.client.jakarta.jms for jakarta. Reading it is a lookup against an in memory map, not an MQI network call.

It re-reads the value on every use rather than caching it. IBM refreshes the resolved connection properties after an automatic client reconnect, which can land on a different queue manager, so a cached value goes silently wrong. It trims the value, since the underlying MQI character field is a fixed 48 bytes and space padded.

It also sets messaging.system to ibmmq in place of the jms the JMS instrumentation applies, so IBM MQ traffic is identifiable and lines up with the IBM MQ instrumentation in other languages. I want to flag this one honestly rather than bury it: every messaging instrumentation in this repository sets messaging.system once at instrumenter build time through AttributesExtractor.constant, and I could not find any precedent for a second module overwriting it on a live span. If maintainers would rather that value stay jms, or would rather IBM MQ get its own instrumentation that creates its own spans the way the .NET distribution did, I would like that decided here before the implementation goes further.

The target span comes from SpanKey, checking PRODUCER, then CONSUMER_PROCESS, then CONSUMER_RECEIVE, never from Span.current(), so nothing can land on an unrelated span that happens to be current. order() is 1000, so the messaging span already exists when the advice fires.

Structure is three submodules. ibmmq-common holds the shared code and imports no javax.jms, jakarta.jms, or com.ibm type. ibmmq/javaagent and ibmmq/ibmmq-jakarta/javaagent cover the two clients. The javax and jakarta variants are separate instrumentation modules because muzzle collects references per class, and the two IBM MQ client jars cannot share a classpath, so a leaked reference would fail muzzle for the other client's users.

Covered span shapes are producer publish and asynchronous delivery through a MessageListener registered with setMessageListener. Pull based receive() is covered on the processing span, with one caveat in Additional context.

Describe alternatives you've considered

Ask the queue manager directly with MQINQ, that is MQQueueManager.Inquire of MQCA_Q_MGR_IDENTIFIER. It returns the right value over a network round trip I do not want on a per message path, when the cached property gives the same answer for a map lookup. Reading it once per connection and caching hits the reconnect case above.

Correlate on the queue manager name instead. It appears to work on a single host and breaks in exactly the federated, multi host case that motivates the request.

Enrich in the collector. It has no connection to any queue manager and no way to know which one an application's span belongs to. The identifier only exists in the application process, in the client the application is already using.

Add the attribute to the existing generic JMS instrumentation. That instrumentation is provider neutral, and referencing com.ibm types from it would drag an IBM MQ dependency into the muzzle references of every JMS user.

Have applications set the attribute themselves with a custom SpanProcessor or manual instrumentation. That is the application code change the javaagent exists to avoid, and every application would have to rediscover where the value lives.

For messaging.system, change JmsMessageAttributesGetter.getSystem() to detect the IBM MQ client and return ibmmq from the JMS instrumentation itself. That keeps the value in the extractor where the rest of the repository sets it, but it puts IBM specific knowledge into provider neutral code, which is the same muzzle problem as above.

Additional context

The JMS receive span cannot carry either attribute. JmsReceiveSpanUtil in jms-common-1.1 starts and ends it in one call through InstrumenterUtil.startAndEnd and never makes it current, so no follow-on advice can reach it while it is open. That span shape is off by default, and a message pulled with receive() still gets both attributes on the span where it is processed. I confirmed this behaviour rather than assuming it: with receive telemetry enabled, every receive span keeps messaging.system as jms and carries no queue manager identifier. Covering that shape would need a hook in the JMS instrumentation, which I would rather settle here than in a pull request.

messaging.ibmmq.queue_manager.id and an ibmmq member for the messaging.system enum are proposed to semantic conventions. An earlier attempt, open-telemetry/semantic-conventions#4014, was closed automatically because the messaging area currently has no active SIG, not on the merits of the proposal. Neither the attribute nor the enum value is ratified, which is why all of this stays behind an experimental flag rather than changing default output.

An implementation is already open as #19896. I opened this issue so the design questions above can be discussed in one place, separately from line level review there.

The same attribute and the same messaging.system value are being pursued for the IBM MQ XMS client in open-telemetry/opentelemetry-dotnet-instrumentation#5404, so settling the naming once covers more than the Java agent. Node.js is intended to follow, and its mainstream IBM MQ binding wraps the MQI C client rather than JMS, so it cannot honestly report jms at all.

Tip

React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review implementation PR #19896 and the proposed ibmmq-common, ibmmq/javaagent, and ibmmq/ibmmq-jakarta/javaagent modules first. Trace SpanKey ordering and the javax and jakarta property lookups, then inspect JmsReceiveSpanUtil to confirm the documented receive-span limitation. Done means the design questions are resolved and the experimental attributes behave as specified without changing generic JMS instrumentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.