prometheus-community / prometheus-community/postgres_exporter
Exporter container log of replica instance always reports pg_replication_slots pq: recovery is in progress
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 835
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 10
Description
What did you do?
I have set up a crunchy postgres cluster on my OpenShift cluster with 1 master and 2 replica instances. exporter container is running as sidecar container. All the replicas logs the following error message:
ts=2023-11-16T13:00:00.362Z caller=namespace.go:236 level=info err="Error running query on database \"localhost:5432\": pg_replication_slots pq: recovery is in progress"
ts=2023-11-16T13:00:00.379Z caller=postgres_exporter.go:731 level=error err="queryNamespaceMappings returned 1 errors"
What did you expect to see?
Replication slots on Replicas are always inactive and in recovery mode so I don't expect to see any errors here
What did you see instead? Under which circumstances?
All replicas reports the same messages listed here:
ts=2023-11-16T13:00:00.362Z caller=namespace.go:236 level=info err="Error running query on database \"localhost:5432\": pg_replication_slots pq: recovery is in progress"
ts=2023-11-16T13:00:00.379Z caller=postgres_exporter.go:731 level=error err="queryNamespaceMappings returned 1 errors"
Environment
OpenShift 4.11 on Azure
- System information:
Linux 4.18.0-372.76.1.el8_6.x86_64 x86_64
- postgres_exporter version:
postgres_exporter, version 0.10.1 (branch: HEAD, revision: 6cff384d7433bcb1104efe3b496cd27c0658eb09)
build user: root@eb21848025d7
build date: 20220114-17:20:30
go version: go1.17.6
platform: linux/amd64
- postgres_exporter flags:
- name: CONFIG_DIR
value: /opt/cpm/conf
- name: POSTGRES_EXPORTER_PORT
value: '9187'
- name: PGBACKREST_INFO_THROTTLE_MINUTES
value: '10'
- name: PG_STAT_STATEMENTS_LIMIT
value: '20'
- name: PG_STAT_STATEMENTS_THROTTLE_MINUTES
value: '-1'
- name: EXPORTER_PG_HOST
value: localhost
- name: EXPORTER_PG_PORT
value: '5432'
- name: EXPORTER_PG_DATABASE
value: postgres
- name: EXPORTER_PG_USER
value: ccp_monitoring
- name: EXPORTER_PG_PASSWORD
valueFrom:
secretKeyRef:
name: flexis-io-dev-scm-billing-monitoring
key: password
- PostgreSQL version:
psql (PostgreSQL) 13.6
- Logs:
ts=2023-11-16T13:00:00.362Z caller=namespace.go:236 level=info err="Error running query on database \"localhost:5432\": pg_replication_slots pq: recovery is in progress"
ts=2023-11-16T13:00:00.379Z caller=postgres_exporter.go:731 level=error err="queryNamespaceMappings returned 1 errors"
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading namespace.go around line 236 and postgres_exporter.go around line 731, where the replica query error is logged and propagated. Reproduce the exporter query against a PostgreSQL 13.6 replica in recovery, then trace how pg_replication_slots is handled. Done means replica exporters no longer report this expected recovery condition as an error while primary behavior remains correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, postgresql
- Domain
- databases, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100