NanmiCoder / NanmiCoder/MediaCrawler
爬取知乎,出现playwright.impl.error
Open
Nobody has claimed this yet.
question
- Dominant language
- Python
- Stars
- 65.3k
- Forks
- 12.6k
- PR merge metrics
- No merged PRs in 30d
Description
⚠️ 提交前确认
- [ ✔] 我已经仔细阅读了项目使用过程中的常见问题汇总
- [✔ ] 我已经搜索并查看了已关闭的issues
- [ ✔] 我确认这不是由于滑块验证码、Cookie过期、Cookie提取错误、平台风控等常见原因导致的问题
❓ 问题描述
爬取知乎时,扫码登陆后返回报错playwright.impl.error,已确定不是梯子问题
🔍 使用场景
- 目标平台: 知乎
- 使用功能: 根据关键词爬取帖子
💻 环境信息
- 操作系统: win10
- Python版本: 3.9.21
- 是否使用IP代理: 无
- 是否使用VPN翻墙软件:无
- 目标平台(抖音/小红书/微博等): 知乎
📋 错误日志
(crawlEnv) PS E:\GraProgram\Crawler\MediaCrawler> python main.py
2025-02-21 23:39:32 MediaCrawler INFO (core.py:345) - [ZhihuCrawler.launch_browser] Begin create browser context ...
2025-02-21 23:39:34 MediaCrawler INFO (core.py:316) - [ZhihuCrawler.create_zhihu_client] Begin create zhihu API client ...
2025-02-21 23:39:34 MediaCrawler INFO (client.py:137) - [ZhiHuClient.pong] Begin to pong zhihu...
2025-02-21 23:39:34 MediaCrawler ERROR (client.py:147) - [ZhiHuClient.pong] Ping zhihu failed: SyntaxError: 语法错误, and try to login again...
2025-02-21 23:39:34 MediaCrawler INFO (login.py:58) - [ZhiHu.begin] Begin login zhihu ...
2025-02-21 23:39:34 MediaCrawler INFO (login.py:74) - [ZhiHu.login_by_qrcode] Begin login zhihu by qrcode ...
Traceback (most recent call last):
File "E:\GraProgram\Crawler\MediaCrawler\main.py", line 66, in <module>
asyncio.get_event_loop().run_until_complete(main())
File "E:\anaconda3\envs\crawlEnv\lib\asyncio\base_events.py", line 647, in run_until_complete
return future.result()
File "E:\GraProgram\Crawler\MediaCrawler\main.py", line 56, in main
await crawler.start()
File "E:\GraProgram\Crawler\MediaCrawler\media_platform\zhihu\core.py", line 85, in start
await login_obj.begin()
File "E:\GraProgram\Crawler\MediaCrawler\media_platform\zhihu\login.py", line 60, in begin
await self.login_by_qrcode()
File "E:\GraProgram\Crawler\MediaCrawler\media_platform\zhihu\login.py", line 77, in login_by_qrcode
base64_qrcode_img = await utils.find_qrcode_img_from_canvas(
File "E:\GraProgram\Crawler\MediaCrawler\tools\crawler_util.py", line 68, in find_qrcode_img_from_canvas
canvas = await page.wait_for_selector(canvas_selector)
File "E:\anaconda3\envs\crawlEnv\lib\site-packages\playwright\async_api\_generated.py", line 7786, in wait_for_selector
await self._impl_obj.wait_for_selector(
File "E:\anaconda3\envs\crawlEnv\lib\site-packages\playwright\_impl\_page.py", line 373, in wait_for_selector
return await self._main_frame.wait_for_selector(**locals_to_params(locals()))
File "E:\anaconda3\envs\crawlEnv\lib\site-packages\playwright\_impl\_frame.py", line 323, in wait_for_selector
await self._channel.send("waitForSelector", locals_to_params(locals()))
File "E:\anaconda3\envs\crawlEnv\lib\site-packages\playwright\_impl\_connection.py", line 59, in send
return await self._connection.wrap_api_call(
File "E:\anaconda3\envs\crawlEnv\lib\site-packages\playwright\_impl\_connection.py", line 509, in wrap_api_call
return await cb()
File "E:\anaconda3\envs\crawlEnv\lib\site-packages\playwright\_impl\_connection.py", line 97, in inner_send
result = next(iter(done)).result()
playwright._impl._errors.TimeoutError: Timeout 30000ms exceeded.
📷 错误截图
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the reported command and trace the QR-code login from media_platform/zhihu/login.py into tools/crawler_util.py. Inspect the canvas selector wait that times out, then verify the Zhihu keyword-crawling flow can complete QR login without the Playwright TimeoutError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- playwright, python
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100