apache / apache/iceberg-python

S3FileSystem.close_session raise "attached to a different loop"

未关闭
#2,785 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
1.1k
派生
581
平均合并
1 天 17 小时
30 天内合并 PR
78

描述

### Apache Iceberg version

main branch

### Please describe the bug 🐞

# Description

This error message does not appear consistently. In my case, it is reliably triggered when performing concurrent queries on an Iceberg table, but it does not occur when querying the same table with a single process or when querying other tables.

This error occurs during the execution of weakref when the Python interpreter's cleanup process is triggered.

The error message is as follows

```
Traceback (most recent call last):
File "/home/*/python3.10/site-packages/aiobotocore/session.py", line 36, in __aexit__
await self._client.__aexit__(exc_type, exc_val, exc_tb)
File "/home/*/python3.10/site-packages/aiobotocore/client.py", line 644, in __aexit__
await self._endpoint.http_session.__aexit__(exc_type, exc_val, exc_tb)
File "/home/*/python3.10/site-packages/aiobotocore/httpsession.py", line 109, in __aexit__
await self._exit_stack.aclose()
File "/home/*/python3.10/contextlib.py", line 656, in aclose
await self.__aexit__(None, None, None)
File "/home/*/python3.10/contextlib.py", line 714, in __aexit__
raise exc_details[1]
File "/home/*/python3.10/contextlib.py", line 697, in __aexit__
cb_suppress = await cb(*exc_details)
File "/home/*/python3.10/site-packages/aiohttp/client.py", line 1483, in __aexit__
await self.close()
File "/home/*/python3.10/site-packages/aiohttp/client.py", line 1349, in close
await self._connector.close()
File "/home/*/python3.10/site-packages/aiohttp/connector.py", line 1044, in close
await super().close(abort_ssl=abort_ssl or self._ssl_shutdown_timeout == 0)
RuntimeError: Task > got Future > attached to a different loop
ERROR:asyncio:Task exception was never retrieved
future: exception=RuntimeError("Task > got Future > attached to a different loop")>
```

# Reason

This bug maybe be caused by #2495 . I revert #2495 and it can work normally.

In #2495 :

In FsspecFileIO, the S3FileSystem instance—which carries an event loop—is stored in threading.local(). This very likely (though I currently lack concrete evidence; it may be caused in combination with other logic) leads to the error during garbage collection at program shutdown.

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

贡献指南

这个仓库没有索引到贡献指南

调研方向

首先跟踪 FsspecFileIO 的 threading.local() S3FileSystem 生命周期,以及此处描述的 S3FileSystem.close_session weakref 清理过程。使用对 Iceberg 表的并发查询重现该故障,然后验证解释器关闭时不再引发 attached-to-a-different-loop 错误。

由索引模型根据 Issue 内容生成。

评估

技术栈
aws, python
领域
backend, cloud
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
冷清
描述清晰度
基本清楚
新手友好度
48/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。