Improve efficiency of reading compressed text files
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Reading compressed text files using textio.ReadFromText() is about five times slower compared to the corresponding transform in Java SDK. This is specially noticeable when using DirectRunner. We should look into how we can improve efficiency of this so that Python SDK is more comparable to Java SDK in this regard.
Imported from Jira [BEAM-2531](https://issues.apache.org/jira/browse/BEAM-2531). Original Jira may contain additional context.
Reported by: chamikara.
Contributor guide
Research direction
Start by locating the Python SDK implementation of textio.ReadFromText() and the corresponding Java transform, then reproduce the reported comparison with DirectRunner. Done means identifying and addressing the efficiency gap so compressed text reading in the Python SDK is more comparable to Java.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100