apache / apache/arrow

[Python] Creating first `pyarrow.array` is slow.

Open
#34,994 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

On my machine creating the first arrow array is unexpectedly slow.

## To reproduce

```bash
python3 -c "import pyarrow as pa; import time; t_send = time.perf_counter(); b = pa.array([200]); print(f'send time={(time.perf_counter() - t_send) * 1_000_000}')"
```
result (in micro second):
```bash
send time=188282.6280016161
```

This is equal to 200ms.

While:

```bash
python3 -c "import pyarrow as pa; import time; pa.array([200]); t_send = time.perf_counter(); b = pa.array([200]); print(f'send time={(time.perf_counter() - t_send) * 1_000_000}')"
```
result (in micro second):
```bash
send time=19.691000488819554
```

## Expected behaviour

Not sure why this warmup is required but can we have this warmup within the `__init__` of arrow.

## Version

```
Name: pyarrow
Version: 11.0.0
```

```
Python 3.10.8
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start with the two provided Python commands and compare the first and second pyarrow.array([200]) calls under pyarrow 11.0.0 and Python 3.10.8. Trace the pyarrow.array entry point to identify the startup work, then verify that the initial call no longer incurs the reported warmup latency without regressing later calls.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.