jackc / jackc/pgx

Batch should flush every so often to reduce memory usage

Open
#1,980 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
14.3k
Forks
1.1k
Avg merge
6d 9h
Merged PRs (30d)
11

Description

At present, `sendBatchExtendedWithDescription` (https://github.com/jackc/pgx/blob/master/conn.go#L1157) queues all queries to be written and performs a single flush at the end. This causes the write buffer in the `pgproto3` connection to become unnecessarily large.

Ideally every `x` statements, or once the buffer passes a certain size, it should be flushed to the server. This both allows the server to start processing, and also reduces the client side memory usage.

The libpq client appears to suggest the ideal approach is to be processing results while also queuing queries (https://www.postgresql.org/docs/current/libpq-pipeline-mode.html), however that would be a larger change.

Contributor guide

Open the contributing guide

Research direction

Start with sendBatchExtendedWithDescription in conn.go around line 1157 and inspect how it queues queries and flushes the pgproto3 connection. Determine an appropriate statement count or buffer-size trigger for intermediate flushes. Done means batches no longer wait for one final flush, while the server can begin processing earlier and client-side write-buffer growth is reduced.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgresql
Domain
database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.