apache / apache/iceberg-python
The upload ID may be invalid, or the upload may have been aborted or completed.
- 主要語言
- Python
- 星號
- 1.1k
- 分支
- 581
- 平均合併
- 1 天 23 小時
- 30 天內合併 PR
- 84
描述
### Apache Iceberg version
0.11.0 (latest release)
### Please describe the bug 🐞
Hello,
I am using a setup with pyiceberg and polaris as rest catalog along with Rustfs for S3 filestorage type. When i am getting this error during table commits.
```
OSError: When completing multiple part upload for key 'Senec/Stage/Bronze/Ampace_P4/BmsAmpaceModules/data/month_timestamp=2025-09/bucket_mainControllerSerial=7/bucket_moduleSerial=17/00000-0-ab52d0d3-77d2-4d72-b10f-146e1d40e866.parquet' in bucket 'senec-bucket': AWS Error NO_SUCH_UPLOAD during CompleteMultipartUpload operation: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
```
And here is the entire traceback:
```
Traceback (most recent call last):
File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/scripts/transfer_monthly.py", line 205, in
transfer_lfp_data_for_module(
File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/scripts/transfer_monthly.py", line 147, in transfer_lfp_data_for_module
iceberg_client.write( File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/novum_iceberg/iceberg_client.py", line 428, in write table.append(pa_table)
File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/env/lib/python3.12/site-packages/pyiceberg/table/__init__.py", line 1427, in append
tx.append(df=df, snapshot_properties=snapshot_properties, branch=branch)
File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/env/lib/python3.12/site-packages/pyiceberg/table/__init__.py", line 523, in append data_files = list(
^^^^^
File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/env/lib/python3.12/site-packages/pyiceberg/io/pyarrow.py", line 2875, in _dataframe_to_data_files
yield from write_file(
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/env/lib/python3.12/site-packages/pyiceberg/io/pyarrow.py", line 2666, in write_parquet
with fo.create(overwrite=True) as fos:
File "pyarrow/io.pxi", line 136, in pyarrow.lib.NativeFile.__exit__
File "pyarrow/io.pxi", line 206, in pyarrow.lib.NativeFile.close
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
OSError: When completing multiple part upload for key 'Senec/Stage/Bronze/Ampace_P4/BmsAmpaceModules/data/month_timestamp=2025-09/bucket_mainControllerSerial=7/bucket_moduleSerial=17/00000-0-ab52d0d3-77d2-4d72-b10f-146e1d40e866.parquet' in bucket 'senec-bucket': AWS Error NO_SUCH_UPLOAD during CompleteMultipartUpload operation: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
```
I have tried to find what is causing the issue but with no luck. I would appreciate it if someone can point to me what could be the problem and a way to fix this issue.
I am working on creating a MCE for this.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
貢獻指南
這個儲存庫沒有索引到貢獻指南
研究方向
從 pyiceberg/io/pyarrow.py 中 write_parquet 和 fo.create(...) 附近開始,參考第 2666 行和第 2875 行附近的 traceback 位置。使用 PyIceberg 0.11.0、作為 REST catalog 的 Polaris 以及作為 S3 儲存的 Rustfs 重現此失敗;完成的標準是找出 multipart-upload 的原因,並透過 regression test 或 MCE 記錄或修正此問題。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- aws, python
- 領域
- backend, data-engineering
- Issue 類型
- 缺陷
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 活躍
- 描述清晰度
- 需要釐清
- 新手友好度
- 35/100