NanmiCoder / NanmiCoder/MediaCrawler

[问题]

Open
#721 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug question
Dominant language
Python
Stars
65.3k
Forks
12.6k
PR merge metrics
No merged PRs in 30d

Description

⚠️ 提交前确认

  • [ ✔] 我已经仔细阅读了项目使用过程中的常见问题汇总
  • [✔ ] 我已经搜索并查看了已关闭的issues
  • [ ✔] 我确认这不是由于滑块验证码、Cookie过期、Cookie提取错误、平台风控等常见原因导致的问题

❓ 问题描述

此项目的微博爬取不能爬取指定微博用户ID的内容

🔍 使用场景

  • 目标平台: 微博
  • 使用功能: 用户主页爬取

💻 环境信息

  • 操作系统: windows
  • Python版本:
  • 是否使用IP代理: 否
  • 是否使用VPN翻墙软件:否
  • 目标平台(抖音/小红书/微博等): 微博

📋 错误日志

PS E:\git> uv run main.py --platform wb --lt qrcode --type creator
2025-09-19 10:37:27 MediaCrawler INFO (core.py:76) - [WeiboCrawler] 使用标准模式启动浏览器
2025-09-19 10:37:27 MediaCrawler INFO (core.py:330) - [WeiboCrawler.launch_browser] Begin create browser context ...
2025-09-19 10:37:28 MediaCrawler INFO (core.py:306) - [WeiboCrawler.create_weibo_client] Begin create weibo API client ...
2025-09-19 10:37:28 MediaCrawler INFO (client.py:88) - [WeiboClient.pong] Begin pong weibo...
2025-09-19 10:37:28 httpx INFO (_client.py:1740) - HTTP Request: GET https://m.weibo.cn/api/config "HTTP/1.1 200 OK"
2025-09-19 10:37:28 MediaCrawler ERROR (client.py:96) - [WeiboClient.pong] cookie may be invalid and again login...
2025-09-19 10:37:28 MediaCrawler INFO (login.py:48) - [WeiboLogin.begin] Begin login weibo ...
2025-09-19 10:37:28 MediaCrawler INFO (login.py:78) - [WeiboLogin.login_by_qrcode] Begin login weibo by qrcode ...
2025-09-19 10:37:28 MediaCrawler INFO (crawler_util.py:42) - [find_login_qrcode] get qrcode by url:https://v2.qr.weibo.cn/inf/gen?api_key=a0241ed0d922e7286303ea5
818292a76&data=https%3A%2F%2Fpassport.weibo.cn%2Fsignin%2Fqrcode%2Fscan%3Fqr%3D3YzBozMHoAAN0z7pGxt82gjXuRdT1M_RvBnFyY29kZQ..%26sinainternalbrowser%3Dtopnav%26sho
wmenu%3D0&datetime=1758249448&deadline=0&level=M&logo=https%3A%2F%2Fimg.t.sinajs.cn%2Ft6%2Fstyle%2Fimages%2Findex%2Fweibo-logo.png&output_type=img&redirect=0&sign=5b7943ad6b41a0f476b0c0972c18880d&size=180&start_time=0&title=sso&type=url
2025-09-19 10:37:28 httpx INFO (_client.py:1740) - HTTP Request: GET https://v2.qr.weibo.cn/inf/gen?api_key=a0241ed0d922e7286303ea5818292a76&data=https%3A%2F%2Fp
assport.weibo.cn%2Fsignin%2Fqrcode%2Fscan%3Fqr%3D3YzBozMHoAAN0z7pGxt82gjXuRdT1M_RvBnFyY29kZQ..%26sinainternalbrowser%3Dtopnav%26showmenu%3D0&datetime=1758249448&
deadline=0&level=M&logo=https%3A%2F%2Fimg.t.sinajs.cn%2Ft6%2Fstyle%2Fimages%2Findex%2Fweibo-logo.png&output_type=img&redirect=0&sign=5b7943ad6b41a0f476b0c0972c18880d&size=180&start_time=0&title=sso&type=url "HTTP/1.1 200 OK"
2025-09-19 10:37:28 MediaCrawler INFO (login.py:94) - [WeiboLogin.login_by_qrcode] Waiting for scan code login, remaining time is 20s
2025-09-19 10:37:45 MediaCrawler INFO (login.py:108) - [WeiboLogin.login_by_qrcode] Login successful then wait for 5 seconds redirect ...
2025-09-19 10:37:50 MediaCrawler INFO (core.py:98) - [WeiboCrawler.start] redirect weibo mobile homepage and update cookies on mobile platform
2025-09-19 10:37:52 MediaCrawler INFO (core.py:280) - [WeiboCrawler.get_creators_and_notes] Begin get weibo creators
2025-09-19 10:37:52 httpx INFO (_client.py:1740) - HTTP Request: GET https://m.weibo.cn/u/5533390220 "HTTP/1.1 200 OK"
2025-09-19 10:37:52 MediaCrawler ERROR (client.py:293) - [WeiboClient.get_creator_info_by_id] get containerid failed
Traceback (most recent call last):
  File "E:\git\main.py", line 86, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File "C:\Users\95839\AppData\Roaming\uv\python\cpython-3.9.23-windows-x86_64-none\lib\asyncio\base_events.py", line 647, in run_until_complete
    return future.result()
  File "E:\git\main.py", line 73, in main
    await crawler.start()
  File "E:\git\media_platform\weibo\core.py", line 112, in start
    await self.get_creators_and_notes()
  File "E:\git\media_platform\weibo\core.py", line 282, in get_creators_and_notes
    createor_info_res: Dict = await self.wb_client.get_creator_info_by_id(creator_id=user_id)
  File "E:\git\media_platform\weibo\client.py", line 294, in get_creator_info_by_id
    raise DataFetchError("get containerid failed")
media_platform.weibo.exception.DataFetchError: get containerid failed

📷 错误截图

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with media_platform/weibo/client.py at get_creator_info_by_id, then trace its call from media_platform/weibo/core.py in get_creators_and_notes. Reproduce the issue with uv run main.py --platform wb --lt qrcode --type creator and user ID 5533390220. Done means the specified Weibo user can be crawled without the get containerid failed DataFetchError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.