apache / apache/arrow

feather.read_table 150x slower when reading columns in newer versions

Open
#33,123 12 comments 0 reactions 0 assignees View on GitHub
Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 20h
Merged PRs (30d)
94

Description

### Description

Performance when reading columns using `feather.read_table` on Arrow 7.0.0-9.0.0 is drastically slower than it was in 6.0.0.

Profiling the code below shows that the bottleneck is somewhere in the `read_names` function of `pyarrow._feather.FeatherReader`.

##### Example

Setup code:

```Java

import pandas as pd
from pyarrow import feather

rows, cols = (1_000_000, 10)
data = {f'c{c}': range(rows) for c in range(cols)}
df = pd.DataFrame(data=data)

feather.write_feather(df, 'test.feather', compression="uncompressed")
```

Benchmarks Arrow 9.0.0:

```Java

%timeit feather.read_table('test.feather', memory_map=True)
%timeit feather.read_table('test.feather', columns=list(df.columns), memory_map=True)

> 178 µs ± 1.23 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
33.8 ms ± 964 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

Benchmarks Arrow 6.0.0:

```Java

%timeit feather.read_table('test.feather', memory_map=True)
%timeit feather.read_table('test.feather', columns=list(df.columns), memory_map=True)

> 173 µs ± 2.12 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
224 µs ± 12.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```

**Environment**: python 3.9, ubuntu 20.04
**Reporter**: [Håkon Magne Holmen](https://issues.apache.org/jira/browse/ARROW-17913)
#### Related issues:
- [[C++] Implement a read range process without caching](https://github.com/apache/arrow/issues/33311) (is related to)

**Note**: *This issue was originally created as [ARROW-17913](https://issues.apache.org/jira/browse/ARROW-17913). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Reproduce the benchmark with feather.read_table and compare column selection with and without the columns argument. Start at pyarrow._feather.FeatherReader.read_names and review related issue 33311, which concerns a C++ read-range process. Done means restoring column-read performance near the Arrow 6.0.0 baseline without changing results.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.