apache / apache/iceberg-python

PyArrowFile class is not compatible with ABFS uri syntax

未关闭
#2,698 7 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
1.1k
派生
581
平均合并
1 天 17 小时
30 天内合并 PR
78

描述

### Apache Iceberg version

0.10.0 (latest release)

### Please describe the bug 🐞

Starting from version 20, Pyarrow has support for Azure filesystems.

ABFS URIs have this [format](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction-abfs-uri): abfs[s]://@.dfs.core.windows.net//

But Pyarrow library expects the following path format for Azure: abfs[s]:////.

As you see, the part "@..core.windows.net" prevents users to use pyarrow file io in Azure environment. This issue CAN be fixed in Pyiceberg by removing account_name part.

The proposed [fix](https://github.com/apache/iceberg-python/pull/2683) is just to start a conversation around the issue. I am not 100% sure how and where this should be fixed.

We know similar issues do not occur with Fsspec file io.

### Examples

We have a very basic setup with RestCatalog:

```
def create_iceberg_catalog():
CATALOG_URI = "https://lakehouse.../catalog"

catalog_config = {
"uri": CATALOG_URI,
PY_IO_IMPL: "pyiceberg.io.pyarrow.PyArrowFileIO",
ADLS_ACCOUNT_NAME: "lakehouseaccount",
}

return RestCatalog("lakehouse", **catalog_config)
```

When we create a table "testns.testtable", it is assigned a following location : abfss://lakehouse-azure-bucket@lakehouseaccount.dfs.core.windows.net/testns/testtable

Then, when we try to append data to the table:
```
data = pa.table(
{
"id": pa.array(range(5), type=pa.int32()), # Ensure 'id' is int32 to match Iceberg schema
"value": [random.choice(["Heads", "Tails"]) for _ in range(5)],
}
)
table.append(data)
```

it throws the following exception:
```
OSError: ListBlobsByHierarchy failed for prefix='aip_test[/test_table-xxx/metadata/snap-xxx.avro](https://xxx/test_table-xxx.avro)'. GetFileInfo is unable to determine whether the path exists. Azure Error: [InvalidResourceName] 400 The specified resource name contains invalid characters.
```

This is because exists() method is called:
```
File [~/.official-venvs/amd64.ipykernel-default.master/lib/python3.12/site-packages/pyiceberg/io/pyarrow.py:368](https://xxx/user/nikita-matckevich/.official-venvs/amd64.ipykernel-default.master/lib/python3.12/site-packages/pyiceberg/io/pyarrow.py#line=367), in PyArrowFile.create(self, overwrite)
366 if not overwrite and self.exists() is True:
```

And it expects the uri without "@akehouseaccount.dfs.core.windows.net". When we monkey-patch the PyArrowFile.__init__ everything works fine:
```
PyArrowFile.old_init = PyArrowFile.__init__
def patched_init(self, location: str, path: str, fs: FileSystem, buffer_size: int = ONE_MEGABYTE):
# Call the original __init__ method
self.old_init(location, path, fs, buffer_size)
self._path = remove_section_between_at_and_slash(path)
print("Logging: PyArrowFile initialized")
PyArrowFile.__init__ = patched_init
```

It does not matter how and with which engine the table was created and written before: all pyarrow methods are not working, even those that are on read path, so it will be impossible to scan a non-empty table as well. We tested it by creating a table with fsspec file io and reading it with pyarrow file io.

It is hard to test this behavior with Azurite, because Azurite uris are different and do not contain "@" part.

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

贡献指南

这个仓库没有索引到贡献指南

调研方向

从 pyiceberg/io/pyarrow.py 中的 PyArrowFile.__init__、create 和 exists() 开始,跟踪 ABFSS 位置是如何传递给 PyArrow 的。使用包含账户名称的 ABFS URI 重现该故障,然后验证 PyArrow 的读取和写入操作能够在没有无效 URI 片段的情况下正常工作。

由索引模型根据 Issue 内容生成。

评估

技术栈
azure, python
领域
cloud, data
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
活跃
描述清晰度
基本清楚
新手友好度
62/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。