influxdata / influxdata/telegraf

Binary Parser: compound entry with repeat

Open
#12,450 6 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Use Case

Using a new assignment called `compound`, the binary parser could parse structs that have repeated parts.

For example, the following struct contains a variable length of data readings from sensors to reduce the number of total packets sent.

```c
typedef struct {
float temperature;
float humidity;
unsigned long timestamp;
} sensor_reading_t;

typedef struct {
short header;
int reading_count;
char[] location;
sensor_reading_t data[];
} multiple_data_packet_t;
```

### Expected behavior

Using the binary parser, we can create an entry to parse the entire struct like so:

```toml
[[inputs.socket_listener]]
service_address = "udp://:8094"

endianess = "le"
data_format = "binary"

[[inputs.socket_listener.binary]]
metric_name = "multi-example"
entries = [
{ bits = 16, omit = true },
{ assignment = "tag", name = "location" },
{ assignment = "compound", repeat = {offset = 16, type = "uint32" }, entries = [
{ name = "temperature", type = "float32" },
{ name = "humidity", type = "float32" },
{ type = "unix_ms", assignment = "time" }
]},
]

[inputs.socket_listener.binary.filter]
selection = [{ offset = 0, bits = 16, match = "0xCAFE" }]
```

This is merely an example of what the final syntax could be. After reading the repeat count, the binary parser would write multiple entries at once.

### Actual behavior

Currently, there's no way to parse different lengths of messages using the binary parser. The only possibility is parsing fixed-length structs.

### Additional info

Some considerations that I haven't thoroughly thought about:
1. Handling timestamp per sub-entry or entry?
2. Where to read the repeat from? Right before the repeated entry or using an offset/bits combination?
a. Should the offset use bits or types (uint8/16/32/64)?
3. How would tags outside the repeated entries apply to the inner ones? Maybe all tags outside are copied to the inner entries?

Contributor guide

Open the contributing guide

Research direction

Start with the binary parser used by the socket_listener input and compare its current fixed-length parsing behavior with the proposed compound entry syntax. Resolve how repeat counts, timestamps, offsets, and outer tags should work, then verify that variable-length repeated sensor readings can be parsed from the example configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.