NanmiCoder / NanmiCoder/MediaCrawler
指定微博平台需要爬取的帖子列表出现Bug,提示Cookie may be invalid and again login
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 65.3k
- Forks
- 12.6k
- PR merge metrics
- No merged PRs in 30d
Description
⚠️ 提交前确认
- 我已经仔细阅读了项目使用过程中的常见问题汇总
- 我已经搜索并查看了已关闭的issues
- 我确认这不是由于滑块验证码、Cookie过期、Cookie提取错误、平台风控等常见原因导致的问题
❓ 问题描述
指定微博平台需要爬取的帖子列表出现Bug,提示Cookie may be invalid and again login。但是我使用search的形式时,却可以正常爬取到数据,只有在指定detail的时候才会出现这个问题
🔍 使用场景
- 目标平台: (如:小红书/抖音/微博等)
微博 - 使用功能: (如:关键词搜索/用户主页爬取等)
根据帖子列表获取相应的评论、点赞等信息
💻 环境信息
- 操作系统:
Win 11 - Python版本:
3.9.6 - 是否使用IP代理:
False - 是否使用VPN翻墙软件:
False - 目标平台(抖音/小红书/微博等):
📋 错误日志
python .\main.py --platform wb --lt cookie --type detail
2025-04-11 19:21:18 MediaCrawler INFO (core.py:313) - [WeiboCrawler.launch_browser] Begin create browser context ...
2025-04-11 19:21:20 MediaCrawler INFO (core.py:276) - [WeiboCrawler.create_weibo_client] Begin create weibo API client ...
2025-04-11 19:21:20 MediaCrawler INFO (client.py:92) - [WeiboClient.pong] Begin pong weibo...
2025-04-11 19:21:20 MediaCrawler ERROR (client.py:100) - [WeiboClient.pong] cookie may be invalid and again login...
2025-04-11 19:21:20 MediaCrawler INFO (login.py:48) - [WeiboLogin.begin] Begin login weibo ...
2025-04-11 19:21:20 MediaCrawler INFO (login.py:116) - [WeiboLogin.login_by_qrcode] Begin login weibo by cookie ...
2025-04-11 19:21:20 MediaCrawler INFO (core.py:86) - [WeiboCrawler.start] redirect weibo mobile homepage and update cookies on mobile platform
2025-04-11 19:21:23 MediaCrawler ERROR (core.py:171) - [WeiboCrawler.get_note_info_task] Get note detail error: get weibo detail err:
2025-04-11 19:21:24 MediaCrawler ERROR (core.py:171) - [WeiboCrawler.get_note_info_task] Get note detail error: get weibo detail err:
2025-04-11 19:21:24 MediaCrawler ERROR (core.py:171) - [WeiboCrawler.get_note_info_task] Get note detail error: get weibo detail err:
2025-04-11 19:21:24 MediaCrawler INFO (core.py:188) - [WeiboCrawler.batch_get_notes_comments] note ids:['5117701752361002', '5045536061393178', '5116304868770360']
2025-04-11 19:21:24 MediaCrawler INFO (core.py:205) - [WeiboCrawler.get_note_comments] begin get note_id: 5117701752361002 comments ...
2025-04-11 19:21:24 MediaCrawler ERROR (core.py:215) - [WeiboCrawler.get_note_comments] may be been blocked, err:Expecting value: line 1 column 1 (char 0)
2025-04-11 19:21:24 MediaCrawler INFO (core.py:205) - [WeiboCrawler.get_note_comments] begin get note_id: 5045536061393178 comments ...
2025-04-11 19:21:24 MediaCrawler ERROR (core.py:215) - [WeiboCrawler.get_note_comments] may be been blocked, err:Expecting value: line 1 column 1 (char 0)
2025-04-11 19:21:24 Me
📷 错误截图
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the reported command python .\main.py --platform wb --lt cookie --type detail and compare it with the working search mode. Start with WeiboCrawler in core.py, WeiboClient in client.py, and the logged detail and comment-fetch paths. Done means detail-mode posts and their comments can be fetched without the cookie or JSON parsing errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100