pgadmin-org / pgadmin-org/pgadmin4

Show transaction log size in Dashboard or Statistics panel.

Open
#6,282 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Feature
Dominant language
Python
Stars
3.8k
Forks
891
Avg merge
4d 7h
Merged PRs (30d)
8

Description

https://dba.stackexchange.com/questions/326988/postgres-with-62-gb-of-table-gobbling-up-over-500-gb-on-disk-solved

Solution (short version):

Coding error in the SQL used by the feeder apps caused "no transaction in progress" warnings (not errors!) that were logged every time an insertion into the tables happened. The log folder ballooned to 400 GB without any indication where the problem was.

Solution (long version):

  • As the SQL used is complex and depends on what type of table is created and which options a user selected, it was created as an array in a step-wise fashion.
  • Each part was tested and copied over into a text editor where it was turned into code for creating the SQL array with a script
  • the resulting code was copied to the IDE for the feeder console apps that process the data and insert the result into the database
  • during one of the copy/paste steps the first line was lost: BEGIN;
  • this resulted in a warning being logged: "no transaction in progress" EVERY TIME an insert happened (about 100 Million to 300 Million times for each data source), and consequently the log folder ballooned to 400 GB
  • the code in the feeder apps deals with database errors but not with warnings
  • pgadmin4 did not show any warning either, nor did it warn about the ballooning log folder
  • as the log folder is locked and inaccessible it was not clear where the hard disk space disappeared to, resulting in this question

Solution posted in the hope my mistake will help others experiencing similar problems. I will also inform the pgadmin team that it might be helpful to keep an eye on size of the log folder.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files, tests, or entry points, so start by locating the Dashboard and Statistics panel implementations and how PostgreSQL log-folder metrics can be accessed. Done means the transaction log size is visible in one of those panels, with behavior and presentation defined and covered by appropriate tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.