apache / apache/beam

[Feature Request]: State performance improvements

Open
#22,964 0 comments 0 reactions 0 assignees View on GitHub
go new feature P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

As follow up to #22736 there are a few things we can do to improve state performance. These are:

1. Caching of state
2. Deferred state reads (readLater and prefetched) - https://beam.apache.org/documentation/programming-guide/#deferred-state-reads

Caching is a larger dig, but readLater and prefetched should be relatively straightforward - they just require running a Get request in parallel with the main computation and adding some thread safety to exec/userstate.go. Prefetch will also require some graph parsing to know when it should perform an early read (see https://docs.google.com/document/d/1rcKa1Z6orDDFr1l8t6NA1eLl6zanQbYAEiAqk39NQUU/edit?disco=AAAAdca3izg for details on how you could do this).

### Issue Priority

Priority: 2

### Issue Component

Component: sdk-go

Contributor guide

Open the contributing guide

Research direction

Start with the state execution and user-state code in exec/userstate.go, then read the deferred state reads section of the Beam programming guide and the linked prefetch design. Implement the readLater and prefetched paths with parallel Get requests and the required thread safety; done means both deferred-read modes work and prefetch can identify earlier reads from the graph.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.