influxdata / influxdata/telegraf

[outputs.postgresql] Telegraf internal error counters are not incremented when errors occur in postgresql output plugin

Open
#17,698 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Relevant telegraf.conf

```toml
[agent]
debug = true
interval = "10s" # default
flush_interval = "10s" # default
round_interval = true # default

[[inputs.internal]]
collect_memstats = true
per_instance = false # default
collect_gostats = false # default

[[inputs.exec]]
commands = [
'''bash -lc 'cat << "EOF"
measurements,device_id=dev1 metric_name="a",metric_value=1.0 1758804689000000000
measurements,device_id=dev1 metric_name="b",metric_value=2.0 1758804690000000000
measurements,device_id=dev1 metric_name="c",metric_value=3.0 1758804691000000000
EOF' '''
]
data_format = "influx"

[[outputs.file]]
files = ["stdout"]

[[outputs.postgresql]]
namepass = ["measurements"]
tagexclude = ["host"]
connection = "host=postgres user=ingester password=ingester_password dbname=pgdb sslmode=disable"
schema = "core"
timestamp_column_name = "observed_at"
timestamp_column_type = "timestamp with time zone"
create_templates = []
add_column_templates = []
```

### Logs from Telegraf

```text
telegraf-1 | 2025-09-25T14:29:48.002754157Z internal_parser,host=978ec7e2d72e,type=influx,version=1.36.1 metrics_parsed=0i,parse_time_ns=0i,errors=0i 1758810580000000000
telegraf-1 | 2025-09-25T14:29:48.002900615Z internal_write,host=978ec7e2d72e,input=exec,version=1.36.1 startup_errors=0i 1758810580000000000
telegraf-1 | 2025-09-25T14:29:48.002928107Z internal_write,host=978ec7e2d72e,input=internal,version=1.36.1 startup_errors=0i 1758810580000000000
telegraf-1 | 2025-09-25T14:29:48.003253390Z internal_gather,host=978ec7e2d72e,input=exec,version=1.36.1 errors=0i,metrics_gathered=0i,gather_time_ns=0i,gather_timeouts=0i 1758810580000000000
telegraf-1 | 2025-09-25T14:29:48.003292504Z internal_write,host=978ec7e2d72e,output=file,version=1.36.1 buffer_limit=10000i,startup_errors=0i,errors=0i,metrics_added=0i,metrics_filtered=0i,buffer_size=0i,write_time_ns=0i,metrics_written=0i,metrics_rejected=0i,metrics_dropped=0i 1758810580000000000
telegraf-1 | 2025-09-25T14:29:48.003310916Z internal_agent,go_version=1.25.1,host=978ec7e2d72e,version=1.36.1 metrics_dropped=0i,metrics_gathered=0i,gather_errors=0i,gather_timeouts=0i,metrics_written=0i,metrics_rejected=0i 1758810580000000000
telegraf-1 | 2025-09-25T14:29:48.003325078Z internal_serializer,host=978ec7e2d72e,type=influx,version=1.36.1 serialization_time_ns=0i,errors=0i,metrics_serialized=0i,bytes_serialized=0i 1758810580000000000
telegraf-1 | 2025-09-25T14:29:48.003346614Z internal_gather,host=978ec7e2d72e,input=internal,version=1.36.1 errors=0i,metrics_gathered=0i,gather_time_ns=0i,gather_timeouts=0i 1758810580000000000
telegraf-1 | 2025-09-25T14:29:48.003389560Z internal_write,host=978ec7e2d72e,output=postgresql,version=1.36.1 metrics_filtered=0i,errors=0i,metrics_dropped=0i,buffer_size=0i,write_time_ns=0i,startup_errors=0i,metrics_added=0i,metrics_written=0i,metrics_rejected=0i,buffer_limit=10000i 1758810580000000000
telegraf-1 | 2025-09-25T14:29:48.003400390Z internal_memstats,host=978ec7e2d72e sys_bytes=27966371i,pointer_lookups=0i,mallocs=119514i,heap_sys_bytes=20152320i,heap_released_bytes=3080192i,heap_objects=69217i,frees=50297i,heap_alloc_bytes=14834296i,heap_idle_bytes=3137536i,heap_in_use_bytes=17014784i,num_gc=2i,alloc_bytes=14834296i,total_alloc_bytes=21300928i 1758810580000000000
telegraf-1 | 2025-09-25T14:29:48.004532527Z measurements,device_id=dev1,host=978ec7e2d72e metric_name="a",metric_value=1 1758804689000000000
telegraf-1 | 2025-09-25T14:29:48.005300185Z measurements,device_id=dev1,host=978ec7e2d72e metric_name="b",metric_value=2 1758804690000000000
telegraf-1 | 2025-09-25T14:29:48.005321137Z measurements,device_id=dev1,host=978ec7e2d72e metric_name="c",metric_value=3 1758804691000000000
telegraf-1 | 2025-09-25T14:29:48.005404197Z 2025-09-25T14:29:48Z D! [outputs.file] Wrote batch of 13 metrics in 6.72655ms
telegraf-1 | 2025-09-25T14:29:48.005874188Z 2025-09-25T14:29:48Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
telegraf-1 | 2025-09-25T14:29:48.075825595Z 2025-09-25T14:29:48Z E! [outputs.postgresql] write error (permanent, dropping sub-batch): metric/table mismatch: Unable to omit field/column from "measurements": critical column "observed_at"
telegraf-1 | 2025-09-25T14:29:48.076803026Z 2025-09-25T14:29:48Z D! [outputs.postgresql] Wrote batch of 3 metrics in 78.744912ms
telegraf-1 | 2025-09-25T14:29:48.076868590Z 2025-09-25T14:29:48Z D! [outputs.postgresql] Buffer fullness: 0 / 10000 metrics
telegraf-1 | 2025-09-25T14:29:57.998786228Z internal_gather,host=978ec7e2d72e,input=internal,version=1.36.1 gather_timeouts=0i,errors=0i,metrics_gathered=10i,gather_time_ns=11526272i 1758810590000000000
telegraf-1 | 2025-09-25T14:29:57.998903570Z internal_write,host=978ec7e2d72e,input=internal,version=1.36.1 startup_errors=0i 1758810590000000000
telegraf-1 | 2025-09-25T14:29:57.998931729Z internal_gather,host=978ec7e2d72e,input=exec,version=1.36.1 errors=0i,metrics_gathered=3i,gather_time_ns=90523695i,gather_timeouts=0i 1758810590000000000
telegraf-1 | 2025-09-25T14:29:57.998875162Z 2025-09-25T14:29:57Z D! [outputs.file] Wrote batch of 13 metrics in 378.475µs
telegraf-1 | 2025-09-25T14:29:57.999065274Z 2025-09-25T14:29:57Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
telegraf-1 | 2025-09-25T14:29:57.998989962Z internal_write,host=978ec7e2d72e,output=file,version=1.36.1 startup_errors=0i,errors=0i,metrics_rejected=0i,metrics_dropped=0i,metrics_added=13i,metrics_written=13i,buffer_limit=10000i,metrics_filtered=0i,buffer_size=0i,write_time_ns=6726550i 1758810590000000000
telegraf-1 | 2025-09-25T14:29:57.999117718Z internal_write,host=978ec7e2d72e,output=postgresql,version=1.36.1 buffer_limit=10000i,metrics_filtered=10i,startup_errors=0i,metrics_added=3i,metrics_written=3i,metrics_rejected=0i,buffer_size=0i,errors=0i,write_time_ns=78744912i,metrics_dropped=0i 1758810590000000000
telegraf-1 | 2025-09-25T14:29:57.999164829Z internal_agent,go_version=1.25.1,host=978ec7e2d72e,version=1.36.1 gather_errors=0i,gather_timeouts=0i,metrics_written=16i,metrics_rejected=0i,metrics_dropped=0i,metrics_gathered=13i 1758810590000000000
telegraf-1 | 2025-09-25T14:29:57.999202694Z internal_parser,host=978ec7e2d72e,type=influx,version=1.36.1 metrics_parsed=3i,parse_time_ns=2404318i,errors=0i 1758810590000000000
telegraf-1 | 2025-09-25T14:29:57.999269758Z internal_serializer,host=978ec7e2d72e,type=influx,version=1.36.1 errors=0i,metrics_serialized=13i,bytes_serialized=2129i,serialization_time_ns=4771231i 1758810590000000000
telegraf-1 | 2025-09-25T14:29:57.999280963Z internal_write,host=978ec7e2d72e,input=exec,version=1.36.1 startup_errors=0i 1758810590000000000
telegraf-1 | 2025-09-25T14:29:57.999289836Z internal_memstats,host=978ec7e2d72e alloc_bytes=14919296i,total_alloc_bytes=21385928i,mallocs=120519i,frees=50329i,heap_sys_bytes=20119552i,heap_in_use_bytes=17047552i,heap_objects=70190i,sys_bytes=27966371i,pointer_lookups=0i,heap_alloc_bytes=14919296i,heap_idle_bytes=3072000i,heap_released_bytes=3014656i,num_gc=2i 1758810590000000000
telegraf-1 | 2025-09-25T14:29:57.999299333Z measurements,device_id=dev1,host=978ec7e2d72e metric_name="a",metric_value=1 1758804689000000000
telegraf-1 | 2025-09-25T14:29:57.999311621Z measurements,device_id=dev1,host=978ec7e2d72e metric_name="b",metric_value=2 1758804690000000000
telegraf-1 | 2025-09-25T14:29:57.999320035Z measurements,device_id=dev1,host=978ec7e2d72e metric_name="c",metric_value=3 1758804691000000000
telegraf-1 | 2025-09-25T14:29:58.006981362Z 2025-09-25T14:29:58Z E! [outputs.postgresql] write error (permanent, dropping sub-batch): metric/table mismatch: Unable to omit field/column from "measurements": critical column "observed_at"
telegraf-1 | 2025-09-25T14:29:58.007252535Z 2025-09-25T14:29:58Z D! [outputs.postgresql] Wrote batch of 3 metrics in 8.667249ms
telegraf-1 | 2025-09-25T14:29:58.007285026Z 2025-09-25T14:29:58Z D! [outputs.postgresql] Buffer fullness: 0 / 10000 metrics
telegraf-1 | 2025-09-25T14:30:08.003302535Z internal_write,host=978ec7e2d72e,output=postgresql,version=1.36.1 metrics_written=6i,metrics_rejected=0i,metrics_dropped=0i,buffer_size=0i,metrics_filtered=20i,errors=0i,write_time_ns=8667249i,metrics_added=6i,buffer_limit=10000i,startup_errors=0i 1758810600000000000
telegraf-1 | 2025-09-25T14:30:08.003759571Z internal_agent,go_version=1.25.1,host=978ec7e2d72e,version=1.36.1 metrics_dropped=0i,metrics_gathered=26i,gather_errors=0i,gather_timeouts=0i,metrics_written=32i,metrics_rejected=0i 1758810600000000000
telegraf-1 | 2025-09-25T14:30:08.003801988Z internal_write,host=978ec7e2d72e,input=exec,version=1.36.1 startup_errors=0i 1758810600000000000
telegraf-1 | 2025-09-25T14:30:08.003825029Z internal_serializer,host=978ec7e2d72e,type=influx,version=1.36.1 serialization_time_ns=5038196i,errors=0i,metrics_serialized=26i,bytes_serialized=4307i 1758810600000000000
telegraf-1 | 2025-09-25T14:30:08.003843404Z internal_gather,host=978ec7e2d72e,input=internal,version=1.36.1 metrics_gathered=20i,gather_time_ns=1978272i,gather_timeouts=0i,errors=0i 1758810600000000000
telegraf-1 | 2025-09-25T14:30:08.003862654Z internal_write,host=978ec7e2d72e,input=internal,version=1.36.1 startup_errors=0i 1758810600000000000
telegraf-1 | 2025-09-25T14:30:08.003882445Z internal_gather,host=978ec7e2d72e,input=exec,version=1.36.1 gather_time_ns=83936439i,gather_timeouts=0i,errors=0i,metrics_gathered=6i 1758810600000000000
telegraf-1 | 2025-09-25T14:30:08.003904653Z internal_write,host=978ec7e2d72e,output=file,version=1.36.1 write_time_ns=378475i,startup_errors=0i,metrics_dropped=0i,buffer_size=0i,errors=0i,metrics_added=26i,metrics_written=26i,metrics_rejected=0i,buffer_limit=10000i,metrics_filtered=0i 1758810600000000000
telegraf-1 | 2025-09-25T14:30:08.003977652Z internal_parser,host=978ec7e2d72e,type=influx,version=1.36.1 errors=0i,metrics_parsed=6i,parse_time_ns=2443223i 1758810600000000000
telegraf-1 | 2025-09-25T14:30:08.003683739Z 2025-09-25T14:30:08Z D! [outputs.file] Wrote batch of 13 metrics in 848.615µs
telegraf-1 | 2025-09-25T14:30:08.004128025Z 2025-09-25T14:30:08Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
telegraf-1 | 2025-09-25T14:30:08.004037485Z internal_memstats,host=978ec7e2d72e total_alloc_bytes=21456624i,pointer_lookups=0i,heap_sys_bytes=20152320i,heap_idle_bytes=3055616i,heap_in_use_bytes=17096704i,heap_objects=70897i,sys_bytes=27966371i,mallocs=121256i,frees=50359i,heap_alloc_bytes=14989992i,heap_released_bytes=2981888i,num_gc=2i,alloc_bytes=14989992i 1758810600000000000
telegraf-1 | 2025-09-25T14:30:08.004645644Z measurements,device_id=dev1,host=978ec7e2d72e metric_name="a",metric_value=1 1758804689000000000
telegraf-1 | 2025-09-25T14:30:08.004771018Z measurements,device_id=dev1,host=978ec7e2d72e metric_name="b",metric_value=2 1758804690000000000
telegraf-1 | 2025-09-25T14:30:08.004797226Z measurements,device_id=dev1,host=978ec7e2d72e metric_name="c",metric_value=3 1758804691000000000
telegraf-1 | 2025-09-25T14:30:08.010702531Z 2025-09-25T14:30:08Z E! [outputs.postgresql] write error (permanent, dropping sub-batch): metric/table mismatch: Unable to omit field/column from "measurements": critical column "observed_at"
telegraf-1 | 2025-09-25T14:30:08.010915862Z 2025-09-25T14:30:08Z D! [outputs.postgresql] Wrote batch of 3 metrics in 9.920592ms
telegraf-1 | 2025-09-25T14:30:08.010927570Z 2025-09-25T14:30:08Z D! [outputs.postgresql] Buffer fullness: 0 / 10000 metrics
```

### System info

Telegraf 1.31.6, Ubuntu 24.04, Docker Server 28.4.0, Docker Compose 2.39.4

### Docker

docker-compose.yml

```yaml
services:
postgres:
image: postgres:latest # 17.6-trixie at time of test
environment:
POSTGRES_DB: pgdb
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
ports:
- 5432:5432
volumes:
- ./pg-init.sql:/docker-entrypoint-initdb.d/pg-init.sql
healthcheck:
test: ["CMD-SHELL", "pg_isready -h 127.0.0.1"]
interval: 1m
timeout: 5s
retries: 2
start_period: 3m
start_interval: 5s

telegraf:
image: telegraf:latest # 1.36.1 at time of test
restart: unless-stopped
depends_on:
postgres:
condition: service_healthy
volumes:
- ./telegraf.conf:/etc/telegraf/telegraf.conf
```

pg-init.sql

```sql
create schema core;

create table core.measurements (
observed_at timestamptz NOT NULL
,device_id text NOT NULL
,metric_name text NOT NULL
,metric_value double precision NOT NULL
);

create role ingester login password 'ingester_password';
-- grant usage on schema core to ingester;
-- grant select on core.measurements to ingester;
-- grant insert on core.measurements to ingester;
```

### Steps to reproduce

1. Create the `compose.yml`, `telegraf.conf`, and `pg-init.sql` files provided above.
2. Run `docker compose up -d` to start the containers.
3. Run `docker compose logs -t -f` to view the container logs with timestamps.
4. Note that the postgresql output plugin reports errors (e.g. `E! [outputs.postgresql] write error (permanent, dropping sub-batch): metric/table mismatch: Unable to omit field/column from "measurements": critical column "observed_at"`), but the internal plugin's error counter for the postgresql output plugin isn't incremented (e.g. `internal_write,host=978ec7e2d72e,output=postgresql,version=1.36.1 buffer_limit=10000i,metrics_filtered=10i,startup_errors=0i,metrics_added=3i,metrics_written=3i,metrics_rejected=0i,buffer_size=0i,errors=0i,write_time_ns=78744912i,metrics_dropped=0i 1758810590000000000`).
5. Note that `metrics_written` is being incremented (as shown above), even though none of the `measurements` metrics are being written to the output destination (i.e. the postgres `core.measurements` table).
6. Note that the precise meaning of `metrics_filtered`, `startup_errors`, `metrics_added`, `metrics_written`, `metrics_rejected`, `buffer_size`, `errors`, and `metrics_dropped` is not documented anywhere in the Telegraf Internal plugin [README](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/internal).

### Expected behavior

I expected the `errors` counter and/or `metrics_dropped` and/or `metrics_rejected` counters in the `internal_write,output=postgresql,...` metric to be incremented.

I also expected the `metrics_written` counter not to be incremented (as telegraf was aware that it had failed to write to the output).

That said, I wasn't really sure what to expect, as there are no definitions provided in the [README](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/internal), and previous issues asking for documentation were closed with no public answers provided (e.g. https://github.com/influxdata/telegraf/issues/16809 and https://github.com/influxdata/telegraf/issues/10257)

### Actual behavior

The `errors` counter, `metrics_dropped`, and `metrics_rejected` counters all remained at zero and the `metrics_written` counter was incremented by the number of metrics which were added to the buffer, regardless of whether they were successfully written to the output or not.

### Additional info

Related issues: https://github.com/influxdata/telegraf/issues/16588, https://github.com/influxdata/telegraf/issues/7964, https://github.com/influxdata/telegraf/issues/8607, https://github.com/influxdata/telegraf/issues/16483

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with the provided telegraf.conf, PostgreSQL setup, and outputs.postgresql logs. Trace the postgresql output plugin's error path and internal_write metrics, then verify that the relevant error counter increments when the critical observed_at column error drops a sub-batch.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgresql
Domain
databases, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.