SDF BoundedSource consumer is ont able to split data from the runner
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 204
Description
This is affecting BigQuery jobs that export into a small number of files and have a large overhead per element in the same stage.
This makes stages slow to execute and unable to split work - so jobs end up being very long. Should look into this.
Imported from Jira [BEAM-11317](https://issues.apache.org/jira/browse/BEAM-11317). Original Jira may contain additional context.
Reported by: pabloem.
Contributor guide
Research direction
Start with the SDF BoundedSource and runner splitting behavior described in the issue, then consult the imported Jira ticket BEAM-11317 for additional context. Reproduce the problem with a BigQuery export producing few files and high per-element overhead; done means the affected stage can split work from the runner and avoids excessively long execution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100