apache / apache/arrow

[Python][Parquet][Windows] Using as_py() on pyarrow.lib.Timestamp extremely slow

Open
#45,630 0 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

We are reading data from a Parquet file (as part of a Delta Lake table) using Dataset.read_batches(). The data includes a column in "timestamp" format in Parquet.

We read using the default settings and map to objects, like so:

```python
result_list = []

for batch in dataset.to_batches(
columns=["StartDate", "ChildObjectName", "Generation"]
):
row_list = zip(*batch.columns)
for row in row_list:
result_list.append(dict(start_date=row[0].as_py(), child_object_name=row[1].as_py(), generation=row[2].as_py()))
```

However, this was taking over 20 seconds per ~100,000 batch read size.

Eventually we narrowed it down to `row[0].as_py()`, and substituted `datetime.fromtimestamp(row[0].value / 1_000_000)`, which brought this to under a second per batch.

My hunch is that the conversion is either looking for the existence of Pandas, or perhaps that this is a side effect of Windows not having a timezone "UTC" defined (these are UTC timestamps) and Python falling back to the pytz database each time.

### Component(s)

Python, Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.