NanmiCoder / NanmiCoder/MediaCrawler

指定微博平台需要爬取的帖子列表出现Bug,提示Cookie may be invalid and again login

Open
#583 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Python
Stars
65.3k
Forks
12.6k
PR merge metrics
No merged PRs in 30d

Description

⚠️ 提交前确认

  • 我已经仔细阅读了项目使用过程中的常见问题汇总
  • 我已经搜索并查看了已关闭的issues
  • 我确认这不是由于滑块验证码、Cookie过期、Cookie提取错误、平台风控等常见原因导致的问题

❓ 问题描述

指定微博平台需要爬取的帖子列表出现Bug,提示Cookie may be invalid and again login。但是我使用search的形式时,却可以正常爬取到数据,只有在指定detail的时候才会出现这个问题

🔍 使用场景

  • 目标平台: (如:小红书/抖音/微博等)
    微博
  • 使用功能: (如:关键词搜索/用户主页爬取等)
    根据帖子列表获取相应的评论、点赞等信息

💻 环境信息

  • 操作系统:
    Win 11
  • Python版本:
    3.9.6
  • 是否使用IP代理:
    False
  • 是否使用VPN翻墙软件:
    False
  • 目标平台(抖音/小红书/微博等):

📋 错误日志

python .\main.py --platform wb --lt cookie --type detail
2025-04-11 19:21:18 MediaCrawler INFO (core.py:313) - [WeiboCrawler.launch_browser] Begin create browser context ...
2025-04-11 19:21:20 MediaCrawler INFO (core.py:276) - [WeiboCrawler.create_weibo_client] Begin create weibo API client ...
2025-04-11 19:21:20 MediaCrawler INFO (client.py:92) - [WeiboClient.pong] Begin pong weibo...
2025-04-11 19:21:20 MediaCrawler ERROR (client.py:100) - [WeiboClient.pong] cookie may be invalid and again login...
2025-04-11 19:21:20 MediaCrawler INFO (login.py:48) - [WeiboLogin.begin] Begin login weibo ...
2025-04-11 19:21:20 MediaCrawler INFO (login.py:116) - [WeiboLogin.login_by_qrcode] Begin login weibo by cookie ...
2025-04-11 19:21:20 MediaCrawler INFO (core.py:86) - [WeiboCrawler.start] redirect weibo mobile homepage and update cookies on mobile platform
2025-04-11 19:21:23 MediaCrawler ERROR (core.py:171) - [WeiboCrawler.get_note_info_task] Get note detail error: get weibo detail err:
2025-04-11 19:21:24 MediaCrawler ERROR (core.py:171) - [WeiboCrawler.get_note_info_task] Get note detail error: get weibo detail err:
2025-04-11 19:21:24 MediaCrawler ERROR (core.py:171) - [WeiboCrawler.get_note_info_task] Get note detail error: get weibo detail err:
2025-04-11 19:21:24 MediaCrawler INFO (core.py:188) - [WeiboCrawler.batch_get_notes_comments] note ids:['5117701752361002', '5045536061393178', '5116304868770360']
2025-04-11 19:21:24 MediaCrawler INFO (core.py:205) - [WeiboCrawler.get_note_comments] begin get note_id: 5117701752361002 comments ...
2025-04-11 19:21:24 MediaCrawler ERROR (core.py:215) - [WeiboCrawler.get_note_comments] may be been blocked, err:Expecting value: line 1 column 1 (char 0)
2025-04-11 19:21:24 MediaCrawler INFO (core.py:205) - [WeiboCrawler.get_note_comments] begin get note_id: 5045536061393178 comments ...
2025-04-11 19:21:24 MediaCrawler ERROR (core.py:215) - [WeiboCrawler.get_note_comments] may be been blocked, err:Expecting value: line 1 column 1 (char 0)
2025-04-11 19:21:24 Me

📷 错误截图

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported command python .\main.py --platform wb --lt cookie --type detail and compare it with the working search mode. Start with WeiboCrawler in core.py, WeiboClient in client.py, and the logged detail and comment-fetch paths. Done means detail-mode posts and their comments can be fetched without the cookie or JSON parsing errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.