apache / apache/iceberg-python

When pyiceberg loads Iceberg tables containing large JSON data into memory, memory usage explodes in pyarrow

Đang mở
#3,168 3 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
1.1k
Fork
581
Merge trung bình
1 ngày 17 giờ
Pull request đã merge (30 ngày)
78

Mô tả

### Apache Iceberg version

0.11.0 (latest release)

### Please describe the bug 🐞

We have a table with a column containing large JSON strings (basically think of large `pydantic` validation results). The resulting datafile in Iceberg itself is a roughly 6MB (GZip compressed) Parquet file, but when querying it, the memory consumption goes to ~4GB. (Just for a few seconds, but long enough to cause out-of-memory issues on some systems.)

The issue is that by default `pyarrow` loads the strings per row into memory, which blows up the memory. If we download the datafile and open it directly via `pyarrow` this behaviour can be reproduced.

There are 2 workarounds in `pyarrow`: 1. Don't load the problematic column (given that is possible in your use case) and 2. switch to dictionary-encoding for set column (example snippet below).

```py
from pyarrow.parquet import read_table
table = read_table("datafile.parquet", read_dictionary=["problematic_column"])
```

Issue in `pyiceberg`:
Regardless if you use `table.scan(...).to_arrow()` or `table.scan(...).to_arrow_batch_reader()`, `pyiceberg` has afaik currently no option to specify the dictionary encoding for certain tables, hence `pyarrow` uses the default encoding and the memory usage explodes.

The `to_arrow_batch_reader` does not help here either, because -as per my understanding- in the batch reader of `pyiceberg` each batch represents an individual datafile. Hence, if there is one problematic 6MB datafile, it makes no difference if you use the batch reader or not. I also have the impression that when you iterate over the reader, `pyarrow` has already loaded the parquet file in a separate thread and this is where the memory explosion actually happens.

So the current only workaround in `pyiceberg` is option 1: Don't load the problematic column by specifying the `selected_fields`:

```py
from pyiceberg.catalog import load_catalog

catalog = load_catalog("default")
table = catalog.load_table("your_table")
reader = table.scan(selected_fields=("all", "other", "columns")).to_arrow_batch_reader()
for batch in reader:
...
```

Expected behaviour:
There should be an option somewhere, e.g. in the data_scan to specify for which columns dictionary encoding should be used. This option should be forwarded to `pyarrow` internally somehow, so that `pyarrow` uses less memory.

Remark:
I would not change the default behaviour. It would be just good to have the option to configure the encoding in pyarrow when needed.

This issue is a follow up for https://github.com/apache/iceberg-python/issues/1205

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Hướng nghiên cứu

Start with the table.scan(...).to_arrow() and to_arrow_batch_reader() entry points, then trace how they invoke pyarrow. Compare this with pyarrow.parquet.read_table and its read_dictionary option. Done means callers can configure dictionary encoding for selected columns without changing the default behavior, with the option forwarded during reads.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python
Lĩnh vực
databases
Loại issue
Lỗi
Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức độ hoạt động
Ít trao đổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
48/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.