NanmiCoder / NanmiCoder/MediaCrawler

爬取知乎,出现playwright.impl.error

Open
#553 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Python
Stars
65.3k
Forks
12.6k
PR merge metrics
No merged PRs in 30d

Description

⚠️ 提交前确认

  • [ ✔] 我已经仔细阅读了项目使用过程中的常见问题汇总
  • [✔ ] 我已经搜索并查看了已关闭的issues
  • [ ✔] 我确认这不是由于滑块验证码、Cookie过期、Cookie提取错误、平台风控等常见原因导致的问题

❓ 问题描述

爬取知乎时,扫码登陆后返回报错playwright.impl.error,已确定不是梯子问题

🔍 使用场景

  • 目标平台: 知乎
  • 使用功能: 根据关键词爬取帖子

💻 环境信息

  • 操作系统: win10
  • Python版本: 3.9.21
  • 是否使用IP代理: 无
  • 是否使用VPN翻墙软件:无
  • 目标平台(抖音/小红书/微博等): 知乎

📋 错误日志

(crawlEnv) PS E:\GraProgram\Crawler\MediaCrawler> python main.py                                            
2025-02-21 23:39:32 MediaCrawler INFO (core.py:345) - [ZhihuCrawler.launch_browser] Begin create browser context ...
2025-02-21 23:39:34 MediaCrawler INFO (core.py:316) - [ZhihuCrawler.create_zhihu_client] Begin create zhihu API client ...
2025-02-21 23:39:34 MediaCrawler INFO (client.py:137) - [ZhiHuClient.pong] Begin to pong zhihu...
2025-02-21 23:39:34 MediaCrawler ERROR (client.py:147) - [ZhiHuClient.pong] Ping zhihu failed: SyntaxError: 语法错误, and try to login again...
2025-02-21 23:39:34 MediaCrawler INFO (login.py:58) - [ZhiHu.begin] Begin login zhihu ...
2025-02-21 23:39:34 MediaCrawler INFO (login.py:74) - [ZhiHu.login_by_qrcode] Begin login zhihu by qrcode ...
Traceback (most recent call last):
  File "E:\GraProgram\Crawler\MediaCrawler\main.py", line 66, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File "E:\anaconda3\envs\crawlEnv\lib\asyncio\base_events.py", line 647, in run_until_complete
    return future.result()
  File "E:\GraProgram\Crawler\MediaCrawler\main.py", line 56, in main
    await crawler.start()
  File "E:\GraProgram\Crawler\MediaCrawler\media_platform\zhihu\core.py", line 85, in start
    await login_obj.begin()
  File "E:\GraProgram\Crawler\MediaCrawler\media_platform\zhihu\login.py", line 60, in begin
    await self.login_by_qrcode()
  File "E:\GraProgram\Crawler\MediaCrawler\media_platform\zhihu\login.py", line 77, in login_by_qrcode
    base64_qrcode_img = await utils.find_qrcode_img_from_canvas(
  File "E:\GraProgram\Crawler\MediaCrawler\tools\crawler_util.py", line 68, in find_qrcode_img_from_canvas
    canvas = await page.wait_for_selector(canvas_selector)
  File "E:\anaconda3\envs\crawlEnv\lib\site-packages\playwright\async_api\_generated.py", line 7786, in wait_for_selector
    await self._impl_obj.wait_for_selector(
  File "E:\anaconda3\envs\crawlEnv\lib\site-packages\playwright\_impl\_page.py", line 373, in wait_for_selector
    return await self._main_frame.wait_for_selector(**locals_to_params(locals()))
  File "E:\anaconda3\envs\crawlEnv\lib\site-packages\playwright\_impl\_frame.py", line 323, in wait_for_selector
    await self._channel.send("waitForSelector", locals_to_params(locals()))
  File "E:\anaconda3\envs\crawlEnv\lib\site-packages\playwright\_impl\_connection.py", line 59, in send
    return await self._connection.wrap_api_call(
  File "E:\anaconda3\envs\crawlEnv\lib\site-packages\playwright\_impl\_connection.py", line 509, in wrap_api_call
    return await cb()
  File "E:\anaconda3\envs\crawlEnv\lib\site-packages\playwright\_impl\_connection.py", line 97, in inner_send
    result = next(iter(done)).result()
playwright._impl._errors.TimeoutError: Timeout 30000ms exceeded.

📷 错误截图

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the reported command and trace the QR-code login from media_platform/zhihu/login.py into tools/crawler_util.py. Inspect the canvas selector wait that times out, then verify the Zhihu keyword-crawling flow can complete QR login without the Playwright TimeoutError.

Written by the indexing model from the issue text.

Assessment

Tech stack
playwright, python
Domain
backend, data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.