influxdata / influxdata/telegraf
XPath Protobuf Parser incorrectly parses google.protobuf.Timestamp
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Relevant telegraf.conf
```toml
[agent]
hostname = "localhost"
flush_interval = "1s"
[[inputs.mqtt_consumer]]
servers = ["tcp://localhost:1883"]
client_id = "telegraf-test-message-consumer"
topics = ["test/message"]
data_format = "xpath_protobuf"
xpath_protobuf_file = "test.proto"
xpath_protobuf_type = "TestMessage"
[[inputs.mqtt_consumer.xpath]]
field_name_expansion = true
metric_name = "'TestMessage'"
field_selection = "timestamp"
[[processors.converter]]
namepass = ["TestMessage"]
[processors.converter.fields]
# timestamp = ["timestamp"]
# timestamp_format = "unix_ns"
[[outputs.file]]
files = ["stdout"]
data_format = "json"
```
### Logs from Telegraf
```text
{"fields":{"timestamp":"5690660001749138879"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138879} 👀
{"fields":{"timestamp":"1749138880574789000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138880}
{"fields":{"timestamp":"1749138881581062000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138881}
{"fields":{"timestamp":"1749138882587107000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138882}
{"fields":{"timestamp":"1749138883593769000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138883}
{"fields":{"timestamp":"1749138884600063000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138884}
{"fields":{"timestamp":"6083220001749138885"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138885} 👀
{"fields":{"timestamp":"1749138886616430000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138886}
{"fields":{"timestamp":"1749138887625780000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138887}
{"fields":{"timestamp":"1749138888632401000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138888}
{"fields":{"timestamp":"1749138889644017000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138889}
{"fields":{"timestamp":"1749138890654236000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138890}
{"fields":{"timestamp":"1749138891664063000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138891}
{"fields":{"timestamp":"1749138892677200000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138892}
{"fields":{"timestamp":"1749138893684626000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138893}
{"fields":{"timestamp":"1749138894695863000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138894}
{"fields":{"timestamp":"7017550001749138895"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138895} 👀
{"fields":{"timestamp":"1749138896715375000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138896}
{"fields":{"timestamp":"1749138906859505000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138906}
{"fields":{"timestamp":"1749138907867704000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138907}
{"fields":{"timestamp":"1749138908873501000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138908}
{"fields":{"timestamp":"1749138909879589000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138909}
{"fields":{"timestamp":"1749138910886459000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138910}
{"fields":{"timestamp":"8925670001749138911"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138911} 👀
{"fields":{"timestamp":"1749138912898451000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138912}
{"fields":{"timestamp":"1749138913904408000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138913}
{"fields":{"timestamp":"9107130001749138914"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138914} 👀
{"fields":{"timestamp":"1749138915917619000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138915}
{"fields":{"timestamp":"1749138916924037000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138916}
{"fields":{"timestamp":"9304070001749138917"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138917} 👀
{"fields":{"timestamp":"1749138918937308000"},"name":"stats","tags":{"topic":"test/message"},"timestamp":1749138918}
```
### System info
Telegraf 1.34.4, WSL: Ubuntu-24.04
### Docker
_No response_
### Steps to reproduce
Ref: https://protobuf.dev/reference/protobuf/google.protobuf/#timestamp
1. Generate proto message with current time in python and send as payload in MQTT
```
msg = TestMessage()
msg.timestamp.GetCurrentTime()
```
2. Start telegraf with config
3. Log data and notice some timestamps are rolled over e.g. "timestamp":"5690660001749138879"
### Expected behavior
e.g. "timestamp":"1749138880574789000"
### Actual behavior
"timestamp":"5690660001749138879" (sometimes because I assume the nano second affects this)
### Additional info
Proto file
```
edition = "2023";
import "google/protobuf/timestamp.proto";
message TestMessage {
google.protobuf.Timestamp timestamp = 1;
}
```
Contributor guide
Research direction
Start with the mqtt_consumer entry point and its xpath_protobuf data format, using the supplied test.proto and Telegraf configuration to reproduce the incorrect google.protobuf.Timestamp values. Compare the parser output with the expected nanosecond timestamp; the work is done when valid Timestamp values remain unchanged across the reported nanosecond cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100