Accessing replies in pipeline mode without a sync
- Dominant language
- Go
- Stars
- 14.3k
- Forks
- 1.1k
- Avg merge
- 6d 9h
- Merged PRs (30d)
- 11
Description
I am using pipeline mode for processing large amounts of data. I have no need for any synchronisation point until the end, and would like to avoid it if possible. However, I do want to see the results as they arrive. This does not seem to be possible currently because results appear to be buffered until Sync is called. Although the results are arriving over the network, they and are being read, they are being buffered by BGReader and are not available to users of Pipeline.
Firstly, is my understanding correct? And secondly, would it make sense to somehow provide access to results as they arrive, to minimise the need for buffering and the need to call Sync?
Contributor guide
Research direction
Start by tracing BGReader and Pipeline to understand how replies are read, buffered, and exposed around Sync. Compare the reported network arrival with when results become available to Pipeline users; done would require a decided API design for accessing replies without a synchronization point.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, postgresql
- Domain
- api, database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100