爬取完一个网站再爬另一个网站总是报数据库数据库连接池关闭时的异常怎么回事

⚠️ 提交前确认

• 我已经仔细阅读了项目使用过程中的常见问题汇总
• 我已经搜索并查看了已关闭的issues
• 我确认这不是由于滑块验证码、Cookie过期、Cookie提取错误、平台风控等常见原因导致的问题

❓ 问题描述

爬取完抖音，再去爬取其他网站比如说微博，总是报数据库连接池关闭时的异常，怎么解决

🔍 使用场景

• 目标平台: (如：小红书/抖音/微博等)
• 使用功能: (如：关键词搜索/用户主页爬取等)

💻 环境信息

• 操作系统: win11
• Python版本: 3.11.5
• 是否使用IP代理: 否
• 是否使用VPN翻墙软件：否
• 目标平台(抖音/小红书/微博等): 抖音

📋 错误日志

Traceback (most recent call last):
File "D:\Code\MediaCrawler\MediaCrawler-main\main.py", line 78, in
asyncio.get_event_loop().run_until_complete(main())
File "D:\Tools\Python\Lib\asyncio\base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "D:\Code\MediaCrawler\MediaCrawler-main\main.py", line 65, in main
await crawler.start()
File "D:\Code\MediaCrawler\MediaCrawler-main\media_platform\weibo\core.py", line 118, in start
await self.search()
File "D:\Code\MediaCrawler\MediaCrawler-main\media_platform\weibo\core.py", line 171, in search
search_res = await self.wb_client.get_note_by_keyword(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Code\MediaCrawler\MediaCrawler-main\media_platform\weibo\client.py", line 131, in get_note_by_keyword
return await self.get(uri, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Code\MediaCrawler\MediaCrawler-main\media_platform\weibo\client.py", line 83, in get
return await self.request(method="GET", url=f"{self.host}{final_uri}", headers=headers, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Code\MediaCrawler\MediaCrawler-main\media_platform\weibo\client.py", line 64, in request
data: Dict = response.json()
^^^^^^^^^^^^^^^
File "D:\Code\MediaCrawler\MediaCrawler-main\venv\Lib\site-packages\httpx_models.py", line 756, in json
return jsonlib.loads(self.text, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Tools\Python\Lib\json_init.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Tools\Python\Lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Tools\Python\Lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\Code\MediaCrawler\MediaCrawler-main\main.py", line 80, in
cleanup()
File "D:\Code\MediaCrawler\MediaCrawler-main\main.py", line 73, in cleanup
asyncio.run(db.close())
File "D:\Tools\Python\Lib\asyncio\runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "D:\Tools\Python\Lib\asyncio\runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Tools\Python\Lib\asyncio\base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "D:\Code\MediaCrawler\MediaCrawler-main\db.py", line 90, in close
db_pool: aiomysql.Pool = db_conn_pool_var.get()
^^^^^^^^^^^^^^^^^^^^^^
LookupError: <ContextVar name='db_conn_pool_var' at 0x0000021DAE586480>
2025-07-26 11:43:55 asyncio ERROR (base_events.py:1771) - Task was destroyed but it is pending!
task: <Task cancelling name='Task-1' coro=<ExpiringLocalCache._start_clear_cron() running at D:\Code\MediaCrawler\MediaCrawler-main\cache\local_cache.py:119> wait_for=>
2025-07-26 11:43:55 asyncio ERROR (base_events.py:1771) - Task was destroyed but it is pending!
task: <Task cancelling name='Task-2' coro=<ExpiringLocalCache._start_clear_cron() running at D:\Code\MediaCrawler\MediaCrawler-main\cache\local_cache.py:119> wait_for=>
Exception ignored in: <function Connection.del at 0x0000021DAE569DA0>
Traceback (most recent call last):
File "D:\Code\MediaCrawler\MediaCrawler-main\venv\Lib\site-packages\aiomysql\connection.py", line 1131, in del
File "D:\Code\MediaCrawler\MediaCrawler-main\venv\Lib\site-packages\aiomysql\connection.py", line 339, in close
File "D:\Tools\Python\Lib\asyncio\proactor_events.py", line 109, in close
File "D:\Tools\Python\Lib\asyncio\base_events.py", line 761, in call_soon
File "D:\Tools\Python\Lib\asyncio\base_events.py", line 519, in _check_closed
RuntimeError: Event loop is closed

📷 错误截图

Start with main.py cleanup(), db.py close(), and the initialization and use of db_conn_pool_var. Reproduce the sequence by crawling Douyin and then Weibo, and inspect the Weibo request path in media_platform/weibo/client.py together with cache/local_cache.py. Done means the original JSONDecodeError is handled without a secondary pool or event-loop cleanup exception when switching platforms.