apache / apache/iceberg-python
S3FileSystem.close_session raise "attached to a different loop"
還沒有人認領這個 Issue。
- 主要語言
- Python
- 星號
- 1.1k
- 分支
- 588
- 平均合併
- 1 天 23 小時
- 30 天內合併 PR
- 84
描述
### Apache Iceberg version
main branch
### Please describe the bug 🐞
# Description
This error message does not appear consistently. In my case, it is reliably triggered when performing concurrent queries on an Iceberg table, but it does not occur when querying the same table with a single process or when querying other tables.
This error occurs during the execution of weakref when the Python interpreter's cleanup process is triggered.
The error message is as follows
```
Traceback (most recent call last):
File "/home/*/python3.10/site-packages/aiobotocore/session.py", line 36, in __aexit__
await self._client.__aexit__(exc_type, exc_val, exc_tb)
File "/home/*/python3.10/site-packages/aiobotocore/client.py", line 644, in __aexit__
await self._endpoint.http_session.__aexit__(exc_type, exc_val, exc_tb)
File "/home/*/python3.10/site-packages/aiobotocore/httpsession.py", line 109, in __aexit__
await self._exit_stack.aclose()
File "/home/*/python3.10/contextlib.py", line 656, in aclose
await self.__aexit__(None, None, None)
File "/home/*/python3.10/contextlib.py", line 714, in __aexit__
raise exc_details[1]
File "/home/*/python3.10/contextlib.py", line 697, in __aexit__
cb_suppress = await cb(*exc_details)
File "/home/*/python3.10/site-packages/aiohttp/client.py", line 1483, in __aexit__
await self.close()
File "/home/*/python3.10/site-packages/aiohttp/client.py", line 1349, in close
await self._connector.close()
File "/home/*/python3.10/site-packages/aiohttp/connector.py", line 1044, in close
await super().close(abort_ssl=abort_ssl or self._ssl_shutdown_timeout == 0)
RuntimeError: Task > got Future > attached to a different loop
ERROR:asyncio:Task exception was never retrieved
future: exception=RuntimeError("Task > got Future > attached to a different loop")>
```
# Reason
This bug maybe be caused by #2495 . I revert #2495 and it can work normally.
In #2495 :
In FsspecFileIO, the S3FileSystem instance—which carries an event loop—is stored in threading.local(). This very likely (though I currently lack concrete evidence; it may be caused in combination with other logic) leads to the error during garbage collection at program shutdown.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
貢獻指南
這個儲存庫沒有索引到貢獻指南
從這裡開始
- 先讀完整個 Issue,再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫,在一個分支上完成修改。
- 送出 Pull Request,並在描述裡引用這個 Issue 編號。
研究方向
首先追蹤 FsspecFileIO 的 threading.local() S3FileSystem 生命週期,以及此處所述的 S3FileSystem.close_session weakref 清理流程。使用對 Iceberg 表的並行查詢重現此失敗,然後驗證直譯器關閉時不再引發 attached-to-a-different-loop 錯誤。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- aws, python
- 領域
- backend, cloud
- Issue 類型
- 缺陷
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 冷清
- 描述清晰度
- 基本清楚
- 新手友好度
- 48/100