apache / apache/beam

Ensure coders are updatable partially in a cluster (aka add versions to coders)

Open
#18,645 2 comments 0 reactions 0 assignees View on GitHub
beam-model improvement P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 5h
Merged PRs (30d)
204

Description

Follow up ticket for the following discussion: https://lists.apache.org/thread.html/f3b2daa740075cc39dc04f08d1eaacfcc2d550cecc04e27be024cf52@%3Cdev.beam.apache.org%3E

Long story short: seems being able to update a live stream is a constraint beam must address. This implies no change in coder serialization. To do it the only way is to put a marker (like a version) otherwise the live update will be broken if part of a cluster is updated only.

This task is about ensuring we support it.

Imported from Jira [BEAM-3616](https://issues.apache.org/jira/browse/BEAM-3616). Original Jira may contain additional context.
Reported by: romain.manni-bucau.

Contributor guide

Open the contributing guide

Research direction

Start with the linked mailing-list discussion and the original Jira ticket BEAM-3616, since this issue provides no files or tests. Determine the design needed to support live updates when only part of a cluster is updated, while preserving coder serialization; done means partial cluster updates remain compatible.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.