influxdata / influxdata/telegraf
outputs.parquet: fields that first appear after the initial flush are silently dropped, with no warning
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Relevant telegraf.conf
```toml
[agent]
interval = "3s"
flush_interval = "3s"
omit_hostname = true
[[inputs.exec]]
commands = ["/path/to/emit.sh"]
data_format = "influx"
[[outputs.parquet]]
directory = "/path/to/out"
`emit.sh` — emits two fields for the first two calls, three thereafter:
#!/bin/sh
n=$(cat counter 2>/dev/null || echo 0); n=$((n+1)); echo $n > counter
if [ "$n" -le 2 ]; then echo "demo a=1i,b=2i"; else echo "demo a=1i,b=2i,c=3i"; fi
```
### Logs from Telegraf
```text
n/a
```
### System info
Telegraf 1.39.2, Linux 6.18.40.
### Docker
n/a
### Steps to reproduce
`outputs.parquet` builds the Arrow schema from the metrics in the **first flush** and then holds it for the lifetime of the file. Fields that appear later are dropped. The README documents this:
> "if additional fields are present after the first metric flush those fields are
> omitted"
and, more severely:
> "If a metric schema does not match the schema in the file it will be dropped."
In reality, a metric is quietly stripped of the fields after the first flush. An operator has no signal that the file they are accumulating is lossy until they query it and find a column missing, which may be weeks later.
This is easy to hit in ordinary operation, not just in contrived cases:
- a sensor that is absent at boot and appears once its driver loads (adding a hardware sensor to a host does exactly this)
- an input whose fields depend on state — GPU metrics that only appear once a process attaches, ECC counters that only appear post-error
- a plugin added or reconfigured while the current file is still open
- any input where a value is `N/A` on the first scrape and numeric afterwards, since `SetIfUsed`-style helpers omit the field entirely in the former case
### Expected behavior
One of, in rough order of preference:
1. The file's schema is widened to accommodate the new field, with earlier rows null for it.
2. A new file is started when the schema changes, so no data is lost.
3. At minimum, a `W!`-level log line naming the dropped field(s) and metric, once per distinct schema mismatch, so the loss is at least observable.
### Actual behavior
Field `c` was present in 6 of the 8 emitted metrics and is absent from the output entirely:
```console
$ duckdb -c "describe select * from read_parquet('out/*.parquet')"
┌───────────┬─────────┐
│ a │ bigint │
│ b │ bigint │
│ timestamp│ bigint │
└───────────┴─────────┘
$ duckdb -c "select * from read_parquet('out/*.parquet')"
a │ b │ timestamp
────┼─────┼─────────────────────
1 │ 2 │ 1785970881000000000
1 │ 2 │ 1785970884000000000
1 │ 2 │ 1785970887000000000 ← c=3 emitted from here on
1 │ 2 │ 1785970890000000000
1 │ 2 │ 1785970893000000000
1 │ 2 │ 1785970896000000000
1 │ 2 │ 1785970899000000000
1 │ 2 │ 1785970902000000000
```
All 8 rows are present; the column is not. Telegraf logged nothing.
### Additional info
While evaluating this plugin for a host-telemetry pipeline, the other blocker was that there is no partitioning. Output files are named `--.parquet` in a single flat directory, so there is no way to lay out `host=…/date=…` (or any tag-based) directory structure of the kind every Parquet query engine uses for partition pruning — DuckDB, DataFusion, Spark and Athena all read Hive-style partitioning natively, and without it a query over a year of data must open every file.
Contributor guide
Research direction
Start with the outputs.parquet implementation and its README, then reproduce the first-flush schema behavior using the provided telegraf.conf and emit.sh. Determine how later fields are handled and choose one of the stated responses: widen the schema, start a new file, or log each distinct mismatch. Done means later fields are no longer silently lost, with coverage for the changing-schema case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100