apache / apache/iceberg-python

load_table consumes enormous amounts of memory on large metadata file

未关闭
#3,162 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
1.1k
派生
581
平均合并
1 天 13 小时
30 天内合并 PR
76

描述

### Apache Iceberg version

0.11.0 (latest release)

### Please describe the bug 🐞

Apologies, this is a bit of a fuzzy one right now, but I thought reporting it anyway.

**Context:**
We're using Iceberg with AWS Glue and AWS S3 as storage. In S3 there are roughly speaking 3 kinds of files (metadata, manifests, and data files). The first one that is read when loading a table via `catalog.load_table()` is the metadata file. The metadata file contains information on all current* snapshots and schema versions of the table. py-iceberg seems to load these completely into memory.

**Issue:**
As we worked on the Iceberg table, there were a lot of snapshots created over time and with that a lot of schema versions. This led to the latest metadata file to be grow to ~10MB gzip compressed (or ~250MB uncompressed JSON). When we load this table via `catalog.load_table()` it consumes ~4GB of memory (total usage of the python process in memray). This is a lot - especially since we only need the latest snapshot and the respective schema version. (Which is probably true for most users I guess.)

**Semi-Workaround:**
One could try to expire some snapshots, e.g. via Sparks `expire_snapshots` procedure [https://iceberg.apache.org/docs/1.10.0/spark-procedures/#expire_snapshots], but it will not get rid of the old / unused schemas unless you set `clean_expired_metadata` as well (which is only supported since 1.10.x, so relatively new).

**(Preliminary) Root-Cause:**
I believe the issue is that we leverage Pydantic's `model_validate_json` in https://github.com/apache/iceberg-python/blob/44ce51a939ccbacf9c87ce6593ad43a752b0871b/pyiceberg/table/metadata.py#L663, which loads the whole JSON into memory and then we seem to keep the full `TableMetadata` object around.

**Suggestion:**
Would it make sense to parse the JSON not fully into memory and load the needed snapshots and schemas lazy / on demand? (Would be also fine, if that is a configurable option of `catalog.load_table()`)

**Remark:**
Obviously we could blame this on an un-maintained Iceberg table, but I think it would be good for the pyIceberg lib to be robust against such scenarios, hence why I opened the issue.

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

贡献指南

这个仓库没有索引到贡献指南

调研方向

从 pyiceberg/table/metadata.py 大约第 663 行开始,这里 catalog.load_table() 使用 Pydantic 的 model_validate_json,并使用 memray 通过大型元数据文件复现峰值内存使用情况。调查加载期间如何保留 snapshots 和 schemas;完成的标准是大型元数据文件的加载峰值内存显著降低,同时最新的 snapshot 和 schema 仍然可用。

由索引模型根据 Issue 内容生成。

评估

技术栈
aws, python
领域
data-engineering, databases
Issue 类型
缺陷
难度
5/5
预计耗时
一周以上
活跃度
冷清
描述清晰度
需要澄清
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。