elastic / elastic/apm

Java APM agent produces invalid json with infinity values in histogram metrics

Open
#916 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Gherkin
Stars
427
Forks
125
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
In my application I wanted to upgrade to the latest java apm agent version available (to date 1.55.1). But this leads to problems with invalid json around histogram metrics and infinity values. Problem that occurs with histograms:
2025-11-27 09:24:10,260 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.AbstractIntakeApiHandler - Response body: {
"accepted": 329,
"errors": [
{
"message": "decode error: data read error: v2.metricsetRoot.Metricset: v2.metricset.Samples: v2.metricsetSampleValue.Values: []float64: readNumberAsString: invalid number, error found in #10 byte of ...|5105400.0,\"Infinity\"|..., bigger context ...|000.0,3843071682022823400.0,4227378850225105400.0,\"Infinity\"],\"counts\":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,|...",
"document": "{\"metricset\":{\"timestamp\":1764235442476000,\"samples\":{\"process_uptime\":{\"value\":390.378},\"avg_statistics_data\":{\"value\":0.0},\"processing_times_last_5_mins.histogram\":{\"values\":[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,16.0,21.0,26.0,31.0,36.0,41.0,46.0,51.0,56.0,64.0,85.0,106.0,127.0,148.0,169.0,190.0,211.0,232.0,256.0,341.0,426.0,511.0,596.0,681.0,766.0,851.0,936.0,1024.0,1365.0,1706.0,2047.0,2388.0,2729.0,3070.0,3411.0,3752.0,4096.0,5461.0,6826.0,8191.0,9556.0,10921.0,12286.0,13651.0,15016.0,16384.0,21845.0,27306.0,32767.0,38228.0,43689.0,49150.0,54611.0,60072.0,65536.0,87381.0,109226.0,131071.0,152916.0,174761.0,196606.0,218451.0,240296.0,262144.0,349525.0,436906.0,524287.0,611668.0,699049.0,786430.0,873811.0,961192.0,1048576.0,1398101.0,1747626.0,2097151.0,2446676.0,2796201.0,3145726.0,3495251.0,3844776.0,4194304.0,5592405.0,6990506.0,8388607.0,9786708.0,11184809.0,12582910.0,13981011.0,15379112.0,16777216.0,22369621.0,27962026.0,33554431.0,39146836.0,44739241.0,50331646.0,55924051.0,61516456.0,67108864.0,89478485.0,111848106.0,134217727.0,156587348.0,178956969.0,201326590.0,223696211.0,246065832.0,268435456.0,357913941.0,447392426.0,536870911.0,626349396.0,715827881.0,805306366.0,894784851.0,984263336.0,1073741824.0,1431655765.0,1789569706.0,2147483647.0,2505397588.0,2863311529.0,3221225470.0,3579139411.0,3937053352.0,4294967296.0,5726623061.0,7158278826.0,8589934591.0,10021590356.0,11453246121.0,12884901886.0,14316557651.0,15748213416.0,17179869184.0,22906492245.0,28633115306.0,34359738367.0,40086361428.0,45812984489.0,51539607550.0,57266230611.0,62992853672.0,68719476736.0,91625968981.0,114532461226.0,137438953471.0,160345445716.0,183251937961.0,206158430206.0,229064922451.0,251971414696.0,274877906944.0,366503875925.0,458129844906.0,549755813887.0,641381782868.0,733007751849.0,824633720830.0,916259689811.0,1007885658792.0,1099511627776.0,1466015503701.0,1832519379626.0,2199023255551.0,2565527131476.0,2932031007401.0,3298534883326.0,3665038759251.0,4031542635176.0,4398046511104.0,5864062014805.0,7330077518506.0,8796093022207.0,10262108525908.0,11728124029609.0,13194139533310.0,14660155037011.0,16126170540712.0,17592186044416.0,23456248059221.0,29320310074026.0,35184372088831.0,41048434103636.0,46912496118441.0,52776558133246.0,58640620148051.0,64504682162856.0,70368744177664.0,93824992236885.0,117281240296106.0,140737488355327.0,164193736414548.0,187649984473769.0,211106232532990.0,234562480592211.0,258018728651432.0,281474976710656.0,375299968947541.0,469124961184426.0,562949953421311.0,656774945658196.0,750599937895081.0,844424930131966.0,938249922368851.0,1032074914605736.0,1125899906842624.0,1501199875790165.0,1876499844737706.0,2251799813685247.0,2627099782632788.0,3002399751580329.0,3377699720527870.0,3752999689475411.0,4128299658422952.0,4503599627370496.0,6004799503160661.0,7505999378950826.0,9007199254740991.0,10508399130531156.0,12009599006321320.0,13510798882111486.0,15011998757901652.0,16513198633691816.0,18014398509481984.0,24019198012642644.0,30023997515803304.0,36028797018963970.0,4.2033596522124624E16,48038396025285290.0,54043195528445950.0,60047995031606610.0,66052794534767270.0,72057594037927940.0,96076792050570580.0,120095990063213230.0,144115188075855870.0,168134386088498530.0,192153584101141150.0,216172782113783800.0,240191980126426460.0,264211178139069100.0,288230376151711740.0,384307168202282300.0,480383960252852900.0,576460752303423500.0,672537544353994100.0,768614336404564600.0,864691128455135200.0,960767920505705900.0,1056844712556276400.0,1152921504606847000.0,1537228672809129200.0,1921535841011411700.0,2305843009213694000.0,2690150177415976400.0,3074457345618258400.0,3458764513820541000.0,3843071682022823400.0,4227378850225105400.0,\"Infinity\"],\"counts\":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],\"type\":\"histogram\"},\"processing_times_statistics_last_5_mins.histogram\":{\"values\":[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,16.0,21.0,26.0,31.0,36.0,41.0,46.0,51.0,56.0,64.0,85.0,106.0,127.0,148.0,169.0,190.0,211.0,232.0,256.0,341.0,426.0,511.0,596.0,681.0,766.0,851.0,936.0,1024.0,1365.0,1706.0,2047.0,2388.0,2729.0,3070.0,3411.0,3752.0,4096.0,5461.0,6826.0,8191.0,9556.0,10921.0,12286.0,13651.0,15016.0,16384.0,21845.0,27306.0,32767.0,38228.0,43689.0,49150.0,54611.0,60072.0,65536.0,87381.0,109226.0,131071.0,152916.0,174761.0,196606.0,218451.0,240296.0,262144.0,349525.0,436906.0,524287.0,611668.0,699049.0,786430.0,873811.0,961192.0,1048576.0,1398101.0,1747626.0,2097151.0,2446676.0,2796201.0,3145726.0,3495251.0,3844776.0,4194304.0,5592405.0,6990506.0,8388607.0,9786708.0,11184809.0,12582910.0,13981011.0,15379112.0,16777216.0,22369621.0,27962026.0,33554431.0,39146836.0,44739241.0,50331646.0,55924051.0,61516456.0,67108864.0,89478485.0,111848106.0,134217727.0,156587348.0,178956969.0,201326590.0,223696211.0,246065832.0,268435456.0,357913941.0,447392426.0,536870911.0,626349396.0,715827881.0,805306366.0,894784851.0,984263336.0,1073741824.0,1431655765.0,1789569706.0,2147483647.0,2505397588.0,2863311529.0,3221225470.0,3579139411.0,3937053352.0,4294967296.0,5726623061.0,7158278826.0,8589934591.0,10021590356.0,11453246121.0,12884901886.0,14316557651.0,15748213416.0,17179869184.0,22906492245.0,28633115306.0,34359738367.0,40086361428.0,45812984489.0,51539607550.0,57266230611.0,62992853672.0,68719476736.0,91625968981.0,114532461226.0,137438953471.0,160345445716.0,183251937961.0,206158430206.0,229064922451.0,251971414696.0,274877906944.0,366503875925.0,458129844906.0,549755813887.0,641381782868.0,733007751849.0,824633720830.0,916259689811.0,1007885658792.0,1099511627776.0,1466015503701.0,1832519379626.0,2199023255551.0,2565527131476.0,2932031007401.0,3298534883326.0,3665038759251.0,4031542635176.0,4398046511104.0,5864062014805.0,7330077518506.0,8796093022207.0,10262108525908.0,11728124029609.0,13194139533310.0,14660155037011.0,16126170540712.0,17592186044416.0,23456248059221.0,29320310074026.0,35184372088831.0,41048434103636.0,46912496118441.0,52776558133246.0,58640620148051.0,64504682162856.0,70368744177664.0,93824992236885.0,117281240296106.0,140737488355327.0,164193736414548.0,187649984473769.0,211106232532990.0,234562480592211.0,258018728651432.0,281474976710656.0,375299968947541.0,469124961184426.0,562949953421311.0,656774945658196.0,750599937895081.0,844424930131966.0,938249922368851.0,1032074914605736.0,1125899906842624.0,1501199875790165.0,1876499844737706.0,2251799813685247.0,2627099782632788.0,3002399751580329.0,3377699720527870.0,3752999689475411.0,4128299658422952.0,4503599627370496.0,6004799503160661.0,7505999378950826.0,9007199254740991.0,10508399130531156.0,12009599006321320.0,13510798882111486.0,15011998757901652.0,16513198633691816.0,18014398509481984.0,24019198012642644.0,30023997515803304.0,36028797018963970.0,4.2033596522124624E16,48038396025285290.0,54043195528445950.0,60047995031606610.0,66052794534767270.0,72057594037927940.0,96076792050570580.0,120095990063213230.0,144115188075855870.0,168134386088498530.0,192153584101141150.0,216172782113783800.0,240191980126426460.0,264211178139069100.0,288230376151711740.0,384307168202282300.0,480383960252852900.0,576460752303423500.0,672537544353994100.0,768614336404564600.0,864691128455135200.0,960767920505705900.0,1056844712556276400.0,1152921504606847000.0,1537228672809129200.0,1921535841011411700.0,2305843009213694000.0,2690150177415976400.0,3074457345618258400.0,3458764513820541000.0,3843071682022823400.0,4227378850225105400.0,\"Infinity\"],\"counts\":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],\"type\":\"histogram\"}}}}"
}
]
}
"Infinity" is coming from the “+Inf” bucket that Micrometer uses for timers/distribution summaries.

JSON does not allow Infinity / NaN as numeric values, and the APM Server’s decoder expects these to be valid float64. As soon as it sees "Infinity" it fails with:

v2.metricsetSampleValue.Values: []float64: readNumberAsString: invalid number

**To Reproduce**
1. Use Java APM Agent version 1.55.1
2. Have histogram percentile publishing enabled in your metric configuration. E.g. like this:
return DistributionSummary.builder(name)
.description(description)
.publishPercentileHistogram()
.publishPercentiles(0.0, 0.99) // min value, 99th percentile
.distributionStatisticExpiry(Duration.ofMinutes(minutesToMonitor))
.register(registry);
3. Then start your application and make sure your metric gets recorded.
4. Error with invalid json will occur.

**Expected behavior**
No invalid json should be produced by apm agent.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.