apache / apache/beam

[Feature Request][Go SDK]: Improve TextIO splitting performance

Open
#25,892 0 comments 0 reactions 0 assignees View on GitHub
go new feature P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

Go TextIO (and future FileIOs) should be updated to perform splitting to match the Java and Python SDF splitting strategies. In particular, have a non-linear chunk size.

See https://github.com/apache/beam/pull/25871/files and https://github.com/apache/beam/blob/abc8099d71b5edad9493c669b5f467e46013b204/sdks/python/apache_beam/io/iobase.py#L894

Specifically:
A simple linear split isn't appropriate for most data processing. Splitting a multi-gigabyte file into a large number of fixed sized chunks can lead to poor performance as each restriction may need to start the file from scratch.

The java and python linked above employ a square root to get a rough "rule of thumb" for splits to avoid over splitting and have more meaningful chunk sizes, and as a fallback, a 64MB chunk size.

From the Java, where the sizes are in bytes:

```
// 1mb --> 1 shard; 1gb --> 32 shards; 1tb --> 1000 shards, 1pb --> 32k shards
desiredChunkSize = Math.max(1 << 20, (long) (1000 * Math.sqrt(estimatedSize)));
```

This allows for larger files to have larger chunks, and reduce overheads related to opening the file repeatedly and similar.

This approach bottoms out at a 1MB chunk size, but 64MB get eight 8MB chunks, while a 6.4GB file gets eighty 80MB chunks, and similar.

Preferably incorporate this as a new method in OffsetRangeRestrictions to make it easier to use across any SDF to determine a good chunk size.

### Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

### Issue Components

- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [X] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Research direction

Start by locating Go TextIO and OffsetRangeRestrictions, then compare the linked Java change with the Python iobase.py splitting logic. Define the reusable chunk-size behavior around the stated square-root rule and 64MB fallback, and verify that large files produce fewer, larger restrictions without changing the intended splitting semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.