elastic / elastic/apm-agent-java

JMX metrics collection is not working (e.g. Hazelcast metrics about maps)

Open
#3,007 3 comments 0 reactions 0 assignees View on GitHub
agent-java bug community
Dominant language
Java
Stars
594
Forks
338
Avg merge
1d 13h
Merged PRs (30d)
25

Description

## Describe the bug

Using the latest version of elastic apm agent (1.36), I can't collect JMX metrics about Hazelcast maps.

> 2023-02-06 14:39:59,140 [hz.cool_zhukovsky.cached.thread-16] WARN co.elastic.apm.agent.jmx.JmxMetricTracker - Can't create metric 'xxx' because attribute 'hits' could not be found.

The problem is created because Hazelcast populates MBean attributes on the fly (see com.hazelcast.internal.metrics.jmx.MetricsMBean). The metrics are not yet ready when the MBeanServerNotification event is fired.

public class MetricsMBean implements DynamicMBean {
private final ConcurrentMap, Type>> metrics = new ConcurrentHashMap<>();

/**
* Adds a metric if necessary and sets its value.
*/
void setMetricValue(String name, String unit, Number value, Type type) {
TriTuple, Type> metricTuple = metrics.get(name);
if (metricTuple == null) {
metricTuple = of(unit, new AtomicReference<>(), type);
metrics.put(name, metricTuple);
}
metricTuple.element2.lazySet(value);
}

void removeMetric(String name) {
metrics.remove(name);
}

@Override
public Object getAttribute(String attributeName) throws AttributeNotFoundException {
TriTuple, Type> attribute = metrics.get(attributeName);
if (attribute == null) {
throw new AttributeNotFoundException(attributeName);
}
return attribute.element2.get();
}

@Override
public AttributeList getAttributes(String[] attributes) {
return Arrays.stream(attributes)
.filter(metrics::containsKey)
.map(a -> uncheckCall(() -> new Attribute(a, getAttribute(a))))
.collect(toCollection(AttributeList::new));
}

}

Other metrics can be properly collected, therefore it is not a problem of agent connectivity or such.

## Steps to reproduce

- Configure metrics export: ELASTIC_APM_CAPTURE_JMX_METRICS=object_name[com.hazelcast:type=Metrics,*,prefix=map,tag0="name=MyMap"] attribute[hits:metric_name=MyMap_hits]
- Create an instance of Hazelcast
- Create a map called `MyMap`
- Add stuff to the map
- See that the warning is produced in the logs

## Expected behaviour

Metrics are reported to Elastic.

**Possible solution:**
- Implement some kind exponential back off and try again registering JMXMetrics later on.

**Improvements:**

- Agent should maybe also listen to AttributeChangeNotification event in addition to MBeanServerNotification (this wouldn't solve the issue of Hazelcast as they don't implement such notification).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.