[Python][Parquet][Windows] Using as_py() on pyarrow.lib.Timestamp extremely slow
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
We are reading data from a Parquet file (as part of a Delta Lake table) using Dataset.read_batches(). The data includes a column in "timestamp" format in Parquet.
We read using the default settings and map to objects, like so:
```python
result_list = []
for batch in dataset.to_batches(
columns=["StartDate", "ChildObjectName", "Generation"]
):
row_list = zip(*batch.columns)
for row in row_list:
result_list.append(dict(start_date=row[0].as_py(), child_object_name=row[1].as_py(), generation=row[2].as_py()))
```
However, this was taking over 20 seconds per ~100,000 batch read size.
Eventually we narrowed it down to `row[0].as_py()`, and substituted `datetime.fromtimestamp(row[0].value / 1_000_000)`, which brought this to under a second per batch.
My hunch is that the conversion is either looking for the existence of Pandas, or perhaps that this is a side effect of Windows not having a timezone "UTC" defined (these are UTC timestamps) and Python falling back to the pytz database each time.
### Component(s)
Python, Parquet
Contributor guide
Assessment
This issue has not been assessed yet.