[Feature Request][Go SDK]: Improve TextIO splitting performance
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What would you like to happen?
Go TextIO (and future FileIOs) should be updated to perform splitting to match the Java and Python SDF splitting strategies. In particular, have a non-linear chunk size.
See https://github.com/apache/beam/pull/25871/files and https://github.com/apache/beam/blob/abc8099d71b5edad9493c669b5f467e46013b204/sdks/python/apache_beam/io/iobase.py#L894
Specifically:
A simple linear split isn't appropriate for most data processing. Splitting a multi-gigabyte file into a large number of fixed sized chunks can lead to poor performance as each restriction may need to start the file from scratch.
The java and python linked above employ a square root to get a rough "rule of thumb" for splits to avoid over splitting and have more meaningful chunk sizes, and as a fallback, a 64MB chunk size.
From the Java, where the sizes are in bytes:
```
// 1mb --> 1 shard; 1gb --> 32 shards; 1tb --> 1000 shards, 1pb --> 32k shards
desiredChunkSize = Math.max(1 << 20, (long) (1000 * Math.sqrt(estimatedSize)));
```
This allows for larger files to have larger chunks, and reduce overheads related to opening the file repeatedly and similar.
This approach bottoms out at a 1MB chunk size, but 64MB get eight 8MB chunks, while a 6.4GB file gets eighty 80MB chunks, and similar.
Preferably incorporate this as a new method in OffsetRangeRestrictions to make it easier to use across any SDF to determine a good chunk size.
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [X] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
Contributor guide
Research direction
Start by locating Go TextIO and OffsetRangeRestrictions, then compare the linked Java change with the Python iobase.py splitting logic. Define the reusable chunk-size behavior around the stated square-root rule and 64MB fallback, and verify that large files produce fewer, larger restrictions without changing the intended splitting semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100