apache / apache/beam

[Bug]: [IcebergIO] Use RecordWriterManager for Partitioned writes

Open
#38,409 0 comments 0 reactions 1 assignee Claimed by @ahmedabu98 View on GitHub
bug io java P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

We should use RecordWriterManager instead of just one RecordWriter to write rows grouped by partition. Doing just one RecordWriter can be dangerous for large writes where one partition has a very large amount of data. Currently, it will lead to one huge file, and the worker will appear idle to the runner the whole time it's writing.

### Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

### Issue Components

- [ ] Component: Python SDK
- [x] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.