prometheus-community / prometheus-community/postgres_exporter

Exporter container log of replica instance always reports pg_replication_slots pq: recovery is in progress

Open
#962 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
3.6k
Forks
835
Avg merge
2d 4h
Merged PRs (30d)
10

Description

What did you do?
I have set up a crunchy postgres cluster on my OpenShift cluster with 1 master and 2 replica instances. exporter container is running as sidecar container. All the replicas logs the following error message:

ts=2023-11-16T13:00:00.362Z caller=namespace.go:236 level=info err="Error running query on database \"localhost:5432\": pg_replication_slots pq: recovery is in progress"
ts=2023-11-16T13:00:00.379Z caller=postgres_exporter.go:731 level=error err="queryNamespaceMappings returned 1 errors"

What did you expect to see?
Replication slots on Replicas are always inactive and in recovery mode so I don't expect to see any errors here

What did you see instead? Under which circumstances?

All replicas reports the same messages listed here:

ts=2023-11-16T13:00:00.362Z caller=namespace.go:236 level=info err="Error running query on database \"localhost:5432\": pg_replication_slots pq: recovery is in progress"
ts=2023-11-16T13:00:00.379Z caller=postgres_exporter.go:731 level=error err="queryNamespaceMappings returned 1 errors"

Environment

OpenShift 4.11 on Azure

  • System information:

Linux 4.18.0-372.76.1.el8_6.x86_64 x86_64

  • postgres_exporter version:

postgres_exporter, version 0.10.1 (branch: HEAD, revision: 6cff384d7433bcb1104efe3b496cd27c0658eb09)
build user: root@eb21848025d7
build date: 20220114-17:20:30
go version: go1.17.6
platform: linux/amd64

  • postgres_exporter flags:
        - name: CONFIG_DIR
          value: /opt/cpm/conf
        - name: POSTGRES_EXPORTER_PORT
          value: '9187'
        - name: PGBACKREST_INFO_THROTTLE_MINUTES
          value: '10'
        - name: PG_STAT_STATEMENTS_LIMIT
          value: '20'
        - name: PG_STAT_STATEMENTS_THROTTLE_MINUTES
          value: '-1'
        - name: EXPORTER_PG_HOST
          value: localhost
        - name: EXPORTER_PG_PORT
          value: '5432'
        - name: EXPORTER_PG_DATABASE
          value: postgres
        - name: EXPORTER_PG_USER
          value: ccp_monitoring
        - name: EXPORTER_PG_PASSWORD
          valueFrom:
            secretKeyRef:
              name: flexis-io-dev-scm-billing-monitoring
              key: password
  • PostgreSQL version:

psql (PostgreSQL) 13.6

  • Logs:
ts=2023-11-16T13:00:00.362Z caller=namespace.go:236 level=info err="Error running query on database \"localhost:5432\": pg_replication_slots pq: recovery is in progress"
ts=2023-11-16T13:00:00.379Z caller=postgres_exporter.go:731 level=error err="queryNamespaceMappings returned 1 errors"

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading namespace.go around line 236 and postgres_exporter.go around line 731, where the replica query error is logged and propagated. Reproduce the exporter query against a PostgreSQL 13.6 replica in recovery, then trace how pg_replication_slots is handled. Done means replica exporters no longer report this expected recovery condition as an error while primary behavior remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgresql
Domain
databases, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.