apache / apache/iceberg-python

The upload ID may be invalid, or the upload may have been aborted or completed.

已关闭
#3,117 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
stale
主要语言
Python
星标
1.1k
派生
581
平均合并
1 天 17 小时
30 天内合并 PR
78

描述

### Apache Iceberg version

0.11.0 (latest release)

### Please describe the bug 🐞

Hello,

I am using a setup with pyiceberg and polaris as rest catalog along with Rustfs for S3 filestorage type. When i am getting this error during table commits.

```
OSError: When completing multiple part upload for key 'Senec/Stage/Bronze/Ampace_P4/BmsAmpaceModules/data/month_timestamp=2025-09/bucket_mainControllerSerial=7/bucket_moduleSerial=17/00000-0-ab52d0d3-77d2-4d72-b10f-146e1d40e866.parquet' in bucket 'senec-bucket': AWS Error NO_SUCH_UPLOAD during CompleteMultipartUpload operation: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
```

And here is the entire traceback:
```
Traceback (most recent call last):
File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/scripts/transfer_monthly.py", line 205, in
transfer_lfp_data_for_module(
File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/scripts/transfer_monthly.py", line 147, in transfer_lfp_data_for_module
iceberg_client.write( File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/novum_iceberg/iceberg_client.py", line 428, in write table.append(pa_table)
File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/env/lib/python3.12/site-packages/pyiceberg/table/__init__.py", line 1427, in append
tx.append(df=df, snapshot_properties=snapshot_properties, branch=branch)
File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/env/lib/python3.12/site-packages/pyiceberg/table/__init__.py", line 523, in append data_files = list(
^^^^^
File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/env/lib/python3.12/site-packages/pyiceberg/io/pyarrow.py", line 2875, in _dataframe_to_data_files
yield from write_file(
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/env/lib/python3.12/site-packages/pyiceberg/io/pyarrow.py", line 2666, in write_parquet
with fo.create(overwrite=True) as fos:
File "pyarrow/io.pxi", line 136, in pyarrow.lib.NativeFile.__exit__
File "pyarrow/io.pxi", line 206, in pyarrow.lib.NativeFile.close
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
OSError: When completing multiple part upload for key 'Senec/Stage/Bronze/Ampace_P4/BmsAmpaceModules/data/month_timestamp=2025-09/bucket_mainControllerSerial=7/bucket_moduleSerial=17/00000-0-ab52d0d3-77d2-4d72-b10f-146e1d40e866.parquet' in bucket 'senec-bucket': AWS Error NO_SUCH_UPLOAD during CompleteMultipartUpload operation: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
```
I have tried to find what is causing the issue but with no luck. I would appreciate it if someone can point to me what could be the problem and a way to fix this issue.

I am working on creating a MCE for this.

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

贡献指南

这个仓库没有索引到贡献指南

调研方向

从 pyiceberg/io/pyarrow.py 中 write_parquet 和 fo.create(...) 附近开始,参考第 2666 行和第 2875 行附近的 traceback 位置。使用 PyIceberg 0.11.0、作为 REST catalog 的 Polaris 以及作为 S3 存储的 Rustfs 重现该失败;完成的标准是确定 multipart-upload 的原因,并通过 regression test 或 MCE 记录或修复该问题。

由索引模型根据 Issue 内容生成。

评估

技术栈
aws, python
领域
backend, data-engineering
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
活跃
描述清晰度
需要澄清
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。