Catch bad FractionConsumed values in the beam SDK.
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
getFractionConsumed in Sources are expected to return values between 0.0 and 1.0
I recently encountered a bug where a bad value of fractionConsumed was sent to a runner. Looking through the beam source, I couldn't find anywhere that we validate the value before we send it (it could be there and I didn't see it, but given that I saw this error with a real source, I don't believe it is.)
I think it'd be useful if the beam SDK caught this error at the time the value is returned and at least generate a useful error in the logs (thus allowing the user to more easily debug the issue)
Imported from Jira [BEAM-1639](https://issues.apache.org/jira/browse/BEAM-1639). Original Jira may contain additional context.
Reported by: sisk.
Contributor guide
Research direction
Start by tracing getFractionConsumed implementations in the Beam SDK and the path that sends the returned value to a runner. Check existing SDK tests around source progress handling, then make invalid values outside 0.0–1.0 produce a useful diagnostic when returned and add coverage demonstrating the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100