[Bug]: unexpected duplicate outputs triggered by quota exceeded exception in finish_bundle()
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
Beam version: 2.46.0
Language: Python
Runner: Dataflow
I am encountering an issue with my BEAM pipeline, which involves performing transformations on a set of documents and saving them into TFRecord. Within the pipeline, I have incorporated a DoFn that logs the transformation info using Stackdriver logging. However, when a quota exceeded exception is raised in the finish_bundle phase of DoFn, the pipeline doesn't fail and some documents end up being saved multiple times, leading to unexpected duplication. This behaviour may be a bug in BEAM or Dataflow.
My current solution is add retry in the finish_bundle whenever quota issue is encountered and this successfully prevents the duplicated docs.
Due to the proprietary restrictions, I am not able to share the source code here. Please let me know if you need more details about this issue.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [X] Component: Google Cloud Dataflow Runner
Contributor guide
Research direction
Start with the Python SDK DoFn finish_bundle() behavior and the Google Cloud Dataflow runner's handling of quota-exceeded exceptions. Reproduce the document transformation pipeline writing TFRecord with logging, then verify that quota failures have consistent failure behavior and do not produce duplicate documents.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100