[Feature Request]: State performance improvements
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What would you like to happen?
As follow up to #22736 there are a few things we can do to improve state performance. These are:
1. Caching of state
2. Deferred state reads (readLater and prefetched) - https://beam.apache.org/documentation/programming-guide/#deferred-state-reads
Caching is a larger dig, but readLater and prefetched should be relatively straightforward - they just require running a Get request in parallel with the main computation and adding some thread safety to exec/userstate.go. Prefetch will also require some graph parsing to know when it should perform an early read (see https://docs.google.com/document/d/1rcKa1Z6orDDFr1l8t6NA1eLl6zanQbYAEiAqk39NQUU/edit?disco=AAAAdca3izg for details on how you could do this).
### Issue Priority
Priority: 2
### Issue Component
Component: sdk-go
Contributor guide
Research direction
Start with the state execution and user-state code in exec/userstate.go, then read the deferred state reads section of the Beam programming guide and the linked prefetch design. Implement the readLater and prefetched paths with parallel Get requests and the required thread safety; done means both deferred-read modes work and prefetch can identify earlier reads from the graph.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100