apache / apache/iceberg-python
The upload ID may be invalid, or the upload may have been aborted or completed.
- 主要言語
- Python
- スター
- 1.1k
- フォーク
- 581
- 平均マージ
- 1日 17時間
- マージ済み PR(30日)
- 77
説明
### Apache Iceberg version
0.11.0 (latest release)
### Please describe the bug 🐞
Hello,
I am using a setup with pyiceberg and polaris as rest catalog along with Rustfs for S3 filestorage type. When i am getting this error during table commits.
```
OSError: When completing multiple part upload for key 'Senec/Stage/Bronze/Ampace_P4/BmsAmpaceModules/data/month_timestamp=2025-09/bucket_mainControllerSerial=7/bucket_moduleSerial=17/00000-0-ab52d0d3-77d2-4d72-b10f-146e1d40e866.parquet' in bucket 'senec-bucket': AWS Error NO_SUCH_UPLOAD during CompleteMultipartUpload operation: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
```
And here is the entire traceback:
```
Traceback (most recent call last):
File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/scripts/transfer_monthly.py", line 205, in
transfer_lfp_data_for_module(
File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/scripts/transfer_monthly.py", line 147, in transfer_lfp_data_for_module
iceberg_client.write( File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/novum_iceberg/iceberg_client.py", line 428, in write table.append(pa_table)
File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/env/lib/python3.12/site-packages/pyiceberg/table/__init__.py", line 1427, in append
tx.append(df=df, snapshot_properties=snapshot_properties, branch=branch)
File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/env/lib/python3.12/site-packages/pyiceberg/table/__init__.py", line 523, in append data_files = list(
^^^^^
File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/env/lib/python3.12/site-packages/pyiceberg/io/pyarrow.py", line 2875, in _dataframe_to_data_files
yield from write_file(
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/parvez/23-0325_senec_monitoring/novum-iceberg/env/lib/python3.12/site-packages/pyiceberg/io/pyarrow.py", line 2666, in write_parquet
with fo.create(overwrite=True) as fos:
File "pyarrow/io.pxi", line 136, in pyarrow.lib.NativeFile.__exit__
File "pyarrow/io.pxi", line 206, in pyarrow.lib.NativeFile.close
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
OSError: When completing multiple part upload for key 'Senec/Stage/Bronze/Ampace_P4/BmsAmpaceModules/data/month_timestamp=2025-09/bucket_mainControllerSerial=7/bucket_moduleSerial=17/00000-0-ab52d0d3-77d2-4d72-b10f-146e1d40e866.parquet' in bucket 'senec-bucket': AWS Error NO_SUCH_UPLOAD during CompleteMultipartUpload operation: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
```
I have tried to find what is causing the issue but with no luck. I would appreciate it if someone can point to me what could be the problem and a way to fix this issue.
I am working on creating a MCE for this.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
調査の方向性
pyiceberg/io/pyarrow.py の write_parquet と fo.create(...) 付近から、2666 行目と 2875 行目付近の traceback の位置を使って調査を開始します。Polaris を REST カタログ、Rustfs を S3 ストレージとして、PyIceberg 0.11.0 で失敗を再現します。multipart-upload の原因を特定し、回帰テストまたは MCE で文書化または修正できれば完了です。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- aws, python
- 領域
- backend, data-engineering
- issue の種類
- バグ
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 活発
- 明瞭さ
- 説明が足りない
- 初心者へのやさしさ
- 35/100