prometheus-community / prometheus-community/postgres_exporter

To many connections open when scrape different instances from one exporter setup

Open
#531 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
3.6k
Forks
835
Avg merge
2d 4h
Merged PRs (30d)
10

Description

What did you do?
Configuring a postgres-exporter instance using docker-compose to scrape metrics from different instances open to many connections

postgres=# select count(*), state, usename from pg_stat_activity group by 2,3;
 count | state  |      usename
-------+--------+-------------------
     4 |        |
     2 | active | replication
     1 | active | postgres
     1 |        | postgres
     2 | idle   | postgres
    50 | idle   | postgres_exporter
(6 rows)

This is the DATASOURCE definition:

environment:
      DATA_SOURCE_NAME: "postgresql://postgres_exporter:XXXX@10.0.2.25:5432/postgres?sslmode=disable,postgresql://postgres_exporter:XXXX@10.0.2.26:5432/postgres?sslmode=disable,postgresql://postgres_exporter:XXXX@10.0.2.27:5432/postgres?sslmode=disable"

docker-compose log:

postgres-exporter    | time="2021-05-10T10:22:14Z" level=info msg="Established new database connection to \"10.0.2.25:5432\"." source="postgres_exporter.go:983"
postgres-exporter    | time="2021-05-10T10:22:14Z" level=info msg="Semantic Version Changed on \"10.0.2.25:5432\": 0.0.0 -> 10.15.0" source="postgres_exporter.go:1552"
postgres-exporter    | time="2021-05-10T10:22:14Z" level=info msg="Established new database connection to \"10.0.2.26:5432\"." source="postgres_exporter.go:983"
postgres-exporter    | time="2021-05-10T10:22:14Z" level=info msg="Semantic Version Changed on \"10.0.2.26:5432\": 0.0.0 -> 10.15.0" source="postgres_exporter.go:1552"
postgres-exporter    | time="2021-05-10T10:22:14Z" level=info msg="Established new database connection to \"10.0.2.27:5432\"." source="postgres_exporter.go:983"
postgres-exporter    | time="2021-05-10T10:22:14Z" level=info msg="Semantic Version Changed on \"10.0.2.27:5432\": 0.0.0 -> 10.15.0" source="postgres_exporter.go:1552"

And if you scrape from only one instance, it does not open such many connections:

environment:
      DATA_SOURCE_NAME: "postgresql://postgres_exporter:XXXX@10.0.2.25:5432/postgres?sslmode=disable"

Connections:

postgres=# select count(*), state, usename from pg_stat_activity group by 2,3;
 count | state  |      usename
-------+--------+-------------------
     4 |        |
     2 | active | replication
     1 | active | postgres
     1 |        | postgres
     2 | idle   | postgres
     1 | idle   | postgres_exporter
(6 rows)

docker-compose log:

postgres-exporter    | time="2021-05-10T10:26:47Z" level=info msg="Established new database connection to \"10.0.2.25:5433\"." source="postgres_exporter.go:983"
postgres-exporter    | time="2021-05-10T10:26:47Z" level=info msg="Semantic Version Changed on \"10.0.2.25:5433\": 0.0.0 -> 10.15.0" source="postgres_exporter.go:1552"

What did you expect to see?

postgres-exporter should not open more than one connection per instance even if you use auto-discover-databases.

Check also https://github.com/prometheus-community/postgres_exporter/issues/436

Environment
This is the docker-compose definition:

version: '3'

services:
  postgres-exporter:
    image: quay.io/prometheuscommunity/postgres-exporter
    container_name: postgres-exporter
    environment:
      DATA_SOURCE_NAME: "postgresql://postgres_exporter:XXXX@10.0.2.25:5432/postgres?sslmode=disable,postgresql://postgres_exporter:XXXX@10.0.2.26:5432/postgres?sslmode=disable,postgresql://postgres_exporter:XXXX@10.0.2.27:5432/postgres?sslmode=disable"
      PG_EXPORTER_EXTEND_QUERY_PATH: "/queries/queries.yaml"
      PG_EXPORTER_AUTO_DISCOVER_DATABASES: "true"
      PG_EXPORTER_EXCLUDE_DATABASES: "template0,template1"
      PG_EXPORTER_CONSTANT_LABELS: "cluster_name=my_cluster"
    volumes:
       - ./postgres-exporter/:/queries
    ports:
      - 9187:9187
    networks:
      - monitor

networks:
  monitor:

  • postgres_exporter version: v0.9.0
$ docker images | grep postgres-exporter
quay.io/prometheuscommunity/postgres-exporter   latest    ca5bb120a572   2 months ago   16.9MB
  • PostgresSQL version:
version                                                                
--------------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 10.15 (Ubuntu 10.15-0ubuntu0.18.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
(1 row)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the connection setup reported at postgres_exporter.go:983 and the auto-discovery configuration in the docker-compose definition, including PG_EXPORTER_AUTO_DISCOVER_DATABASES. Reproduce with the multi-instance DATA_SOURCE_NAME and queries.yaml setup, then verify that scraping leaves no more than one connection per instance; compare against issue #436.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, go, postgresql
Domain
databases, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.