jackc / jackc/pgx

Accessing replies in pipeline mode without a sync

Open
#2,025 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
14.3k
Forks
1.1k
Avg merge
6d 9h
Merged PRs (30d)
11

Description

I am using pipeline mode for processing large amounts of data. I have no need for any synchronisation point until the end, and would like to avoid it if possible. However, I do want to see the results as they arrive. This does not seem to be possible currently because results appear to be buffered until Sync is called. Although the results are arriving over the network, they and are being read, they are being buffered by BGReader and are not available to users of Pipeline.

Firstly, is my understanding correct? And secondly, would it make sense to somehow provide access to results as they arrive, to minimise the need for buffering and the need to call Sync?

Contributor guide

Open the contributing guide

Research direction

Start by tracing BGReader and Pipeline to understand how replies are read, buffered, and exposed around Sync. Compare the reported network arrival with when results become available to Pipeline users; done would require a decided API design for accessing replies without a synchronization point.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgresql
Domain
api, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.