Batch should flush every so often to reduce memory usage
- Dominant language
- Go
- Stars
- 14.3k
- Forks
- 1.1k
- Avg merge
- 6d 9h
- Merged PRs (30d)
- 11
Description
At present, `sendBatchExtendedWithDescription` (https://github.com/jackc/pgx/blob/master/conn.go#L1157) queues all queries to be written and performs a single flush at the end. This causes the write buffer in the `pgproto3` connection to become unnecessarily large.
Ideally every `x` statements, or once the buffer passes a certain size, it should be flushed to the server. This both allows the server to start processing, and also reduces the client side memory usage.
The libpq client appears to suggest the ideal approach is to be processing results while also queuing queries (https://www.postgresql.org/docs/current/libpq-pipeline-mode.html), however that would be a larger change.
Contributor guide
Research direction
Start with sendBatchExtendedWithDescription in conn.go around line 1157 and inspect how it queues queries and flushes the pgproto3 connection. Determine an appropriate statement count or buffer-size trigger for intermediate flushes. Done means batches no longer wait for one final flush, while the server can begin processing earlier and client-side write-buffer growth is reduced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, postgresql
- Domain
- database
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100