Checkpoint support in Beam
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Beam could support checkpoints - similar to:
* flink's snapshots
* scalding's checkpoints
Checkpoint should provides a simple mechanism to read and write intermediate results. It would be useful for debugging one part of a long flow, when you would otherwise have to run many steps to get to the one you care about.
Imported from Jira [BEAM-1458](https://issues.apache.org/jira/browse/BEAM-1458). Original Jira may contain additional context.
Reported by: ravwojdyla.
Contributor guide
Research direction
No files, tests, or entry points are named. Start by reviewing the original Jira issue BEAM-1458 and the referenced Flink snapshots and Scalding checkpoints, then determine the intended read/write behavior for intermediate results and the Beam components it would affect. Done means an agreed checkpoint design with clear scope and validation criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100