Memory leak in Postgres query's process

What did you do?
Run PostgreSQL and postgres_exporter on the same VM (Ubuntu) in Azure. We scrape postgres_exporter endpoint in every 15 sec.

What did you expect to see?
Almost constant and low memory usage at Postgres query's process (which belongs to postgres_exporter).

What did you see instead? Under which circumstances?

As you can seen there is a sawtooth in memory consumption. We started to measure our Linux processes memory usage with htop and pg_top. We found Postgres query's process (which belongs to postgres_exporter) eats more and more memory (at every scraping, in our case 1MB / 1 min). When we stopped our postgres_exporter's service then the issue disappeared. We also tried to stop scraping then memory consumption stay constant. And finally we tried to kill Postgres query's idle process and after that occupied memory has been freed.

Our workaround is based on this information. We wrote a bash script which kills Postgres query's process which belongs to exporter in every 15 minutes and in this case memory usage sawtooth is much smaller.

The largest PostgreSQL database size is around 2Tb. We use the following PG extensions:

• timescaledb 2.7.1
• plpgsql 1.0
• plpython3u 1.0
• postgis 3.2.1
• unaccent 1.1
• pg_trgm 1.6

We turn off any data transformation and loading tasks in our ELT system.

Environment

• System information:

Linux 5.15.0-1038-azure x86_64

• postgres_exporter version:

postgres_exporter, version 0.12.0 (branch: HEAD, revision: f9a1edbbc6d629663aa658378e1d989293e6e488)
build user: root@463311dfdfb6
build date: 20230321-23:52:03
go version: go1.20.2
platform: linux/amd64
tags: netgo static_build

• postgres_exporter flags:

We don't use any flags, we use default settings.

• PostgreSQL version:

PostgreSQL 14.4 (Ubuntu 14.4-1.pgdg20.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, 64-bit

• Logs:
  We haven't got any relevant log.

Start by reproducing the issue with postgres_exporter 0.12.0, PostgreSQL 14.4, and a 15-second scrape interval on Linux. Observe the exporter process and its PostgreSQL idle query while scraping, then narrow the leak to the scrape path. Done means memory remains stable during repeated scrapes and the reported behavior is covered by verification.