influxdata / influxdata/telegraf

telegraf wrongly assumes metric is prometheus histogram and breaks itself

Open
#14,371 6 comments 0 reactions 1 assignee Claimed by @skartikey View on GitHub
bug upstream
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Relevant telegraf.conf

```toml
[global_tags]
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = "0s"
hostname = ""
omit_hostname = false
[[inputs.influxdb_listener]]
service_address = "127.0.0.1:8086"

[[outputs.opentelemetry]]
```

### Logs from Telegraf

```text
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
```

### System info

telegraf 1.28.5

### Docker

_No response_

### Steps to reproduce

1. Send some influxdb metrics that include a `count` field ending in `i`, in example `count=1i`
2. Attempt to export via outputs.opentelemetry
3. Observe telegraf complain about the incorrect assumption it made with the warning message: "[outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64"

### Expected behavior

The metric should be converted to a format that is compatible with opentelemetry's specification and sent to the specified output.

### Actual behavior

telegraf freaks out about a type assumption that it made on it's own and refuses to output via opentelemetry

### Additional info

I think the main problem is that the logic for assuming a prometheus histogram is faulty. This also came up in:
https://github.com/influxdata/telegraf/pull/12431

I would suggest either:

The logic shouldn't be assuming it's a prometheus histogram when the `count` field is of type `int64`.

The value should be converted to a supported type automatically

It should be possible in the configuration file to explicitly define the conversion for values of incompatible types

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.