feather.read_table 150x slower when reading columns in newer versions
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 94
Description
### Description
Performance when reading columns using `feather.read_table` on Arrow 7.0.0-9.0.0 is drastically slower than it was in 6.0.0.
Profiling the code below shows that the bottleneck is somewhere in the `read_names` function of `pyarrow._feather.FeatherReader`.
##### Example
Setup code:
```Java
import pandas as pd
from pyarrow import feather
rows, cols = (1_000_000, 10)
data = {f'c{c}': range(rows) for c in range(cols)}
df = pd.DataFrame(data=data)
feather.write_feather(df, 'test.feather', compression="uncompressed")
```
Benchmarks Arrow 9.0.0:
```Java
%timeit feather.read_table('test.feather', memory_map=True)
%timeit feather.read_table('test.feather', columns=list(df.columns), memory_map=True)
> 178 µs ± 1.23 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
33.8 ms ± 964 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```
Benchmarks Arrow 6.0.0:
```Java
%timeit feather.read_table('test.feather', memory_map=True)
%timeit feather.read_table('test.feather', columns=list(df.columns), memory_map=True)
> 173 µs ± 2.12 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
224 µs ± 12.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```
**Environment**: python 3.9, ubuntu 20.04
**Reporter**: [Håkon Magne Holmen](https://issues.apache.org/jira/browse/ARROW-17913)
#### Related issues:
- [[C++] Implement a read range process without caching](https://github.com/apache/arrow/issues/33311) (is related to)
**Note**: *This issue was originally created as [ARROW-17913](https://issues.apache.org/jira/browse/ARROW-17913). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Reproduce the benchmark with feather.read_table and compare column selection with and without the columns argument. Start at pyarrow._feather.FeatherReader.read_names and review related issue 33311, which concerns a C++ read-range process. Done means restoring column-read performance near the Arrow 6.0.0 baseline without changing results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100