influxdata / influxdata/telegraf
Binary Parser: compound entry with repeat
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Use Case
Using a new assignment called `compound`, the binary parser could parse structs that have repeated parts.
For example, the following struct contains a variable length of data readings from sensors to reduce the number of total packets sent.
```c
typedef struct {
float temperature;
float humidity;
unsigned long timestamp;
} sensor_reading_t;
typedef struct {
short header;
int reading_count;
char[] location;
sensor_reading_t data[];
} multiple_data_packet_t;
```
### Expected behavior
Using the binary parser, we can create an entry to parse the entire struct like so:
```toml
[[inputs.socket_listener]]
service_address = "udp://:8094"
endianess = "le"
data_format = "binary"
[[inputs.socket_listener.binary]]
metric_name = "multi-example"
entries = [
{ bits = 16, omit = true },
{ assignment = "tag", name = "location" },
{ assignment = "compound", repeat = {offset = 16, type = "uint32" }, entries = [
{ name = "temperature", type = "float32" },
{ name = "humidity", type = "float32" },
{ type = "unix_ms", assignment = "time" }
]},
]
[inputs.socket_listener.binary.filter]
selection = [{ offset = 0, bits = 16, match = "0xCAFE" }]
```
This is merely an example of what the final syntax could be. After reading the repeat count, the binary parser would write multiple entries at once.
### Actual behavior
Currently, there's no way to parse different lengths of messages using the binary parser. The only possibility is parsing fixed-length structs.
### Additional info
Some considerations that I haven't thoroughly thought about:
1. Handling timestamp per sub-entry or entry?
2. Where to read the repeat from? Right before the repeated entry or using an offset/bits combination?
a. Should the offset use bits or types (uint8/16/32/64)?
3. How would tags outside the repeated entries apply to the inner ones? Maybe all tags outside are copied to the inner entries?
Contributor guide
Research direction
Start with the binary parser used by the socket_listener input and compare its current fixed-length parsing behavior with the proposed compound entry syntax. Resolve how repeat counts, timestamps, offsets, and outer tags should work, then verify that variable-length repeated sensor readings can be parsed from the example configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100