NanmiCoder / NanmiCoder/MediaCrawler

[问题] 微博detail 的爬取模式中无法爬取

Open
#914 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug question
Dominant language
Python
Stars
65.3k
Forks
12.6k
No merged PRs in 30d

Description

⚠️ 提交前确认

  • 我已经仔细阅读了项目使用过程中的常见问题汇总

  • 我已经搜索并查看了已关闭的issues

  • 我确认这不是由于滑块验证码、Cookie过期、Cookie提取错误、平台风控等常见原因导致的问题

❓ 问题描述

在使用微博detail 的爬取模式中无法爬取
使用的命令：uv run python main.py --platform wb --lt qrcode --type detail --save_data_option excel --specified_id https://weibo.com/7798025479/R3bk1ydAR?refer_flag=1001030103_ --get_comment true --get_sub_comment true --headless false
切换到微博的search模式可以正常爬取

🔍 使用场景

  • 目标平台: (如：小红书/抖音/微博等) 微博
  • 使用功能: (如：关键词搜索/用户主页爬取等) 指定帖子ID列表并爬取帖子与评论

💻 环境信息

  • 操作系统: windows 11
  • Python版本: Python 3.14.5
  • 是否使用IP代理: 无
  • 是否使用VPN翻墙软件：无
  • 目标平台(抖音/小红书/微博等): 微博

📋 错误日志

在此粘贴完整的错误日志

PS C:\Program Files\MediaCrawler-main> uv run python main.py --platform wb --lt qrcode --type detail --save_data_option excel --specified_id https://weibo.com/7798025479/R3bk1ydAR?refer_flag=1001030103_ --get_comment true --get_sub_comment true --headless false
2026-06-08 20:26:35 MediaCrawler INFO (core.py:79) - [WeiboCrawler] Launching browser with CDP mode
2026-06-08 20:26:35 MediaCrawler INFO (cdp_browser.py:152) - [CDPBrowserManager] Connecting to existing browser on port 9222...
2026-06-08 20:26:35 MediaCrawler INFO (cdp_browser.py:155) - [CDPBrowserManager] Make sure remote debugging is enabled in your browser: chrome://inspect/#remote-debugging
2026-06-08 20:26:35 MediaCrawler INFO (cdp_browser.py:163) - [CDPBrowserManager] Waiting up to 60s for browser CDP connection...
2026-06-08 20:26:35 MediaCrawler INFO (cdp_browser.py:237) - [CDPBrowserManager] CDP port 9222 is accessible
2026-06-08 20:26:35 MediaCrawler INFO (cdp_browser.py:323) - [CDPBrowserManager] Connecting to existing browser via CDP: ws://localhost:9222/devtools/browser
2026-06-08 20:26:35 MediaCrawler INFO (cdp_browser.py:324) - [CDPBrowserManager] Please check your browser for a confirmation dialog and accept it
2026-06-08 20:26:42 MediaCrawler INFO (cdp_browser.py:337) - [CDPBrowserManager] Successfully connected to browser
2026-06-08 20:26:42 MediaCrawler INFO (cdp_browser.py:338) - [CDPBrowserManager] Browser contexts count: 1
2026-06-08 20:26:42 MediaCrawler INFO (cdp_browser.py:363) - [CDPBrowserManager] Using existing browser context
2026-06-08 20:26:42 MediaCrawler INFO (cdp_browser.py:194) - [CDPBrowserManager] Successfully connected to existing browser
2026-06-08 20:26:42 MediaCrawler INFO (core.py:411) - [WeiboCrawler] CDP browser info: {'version': '149.0.7827.54', 'contexts_count': 1, 'debug_port': 9222, 'is_connected': True}
2026-06-08 20:26:47 MediaCrawler INFO (core.py:341) - [WeiboCrawler.create_weibo_client] Begin create weibo API client ...
2026-06-08 20:26:47 MediaCrawler INFO (client.py:120) - [WeiboClient.pong] Begin pong weibo...
2026-06-08 20:26:48 MediaCrawler INFO (client.py:277) - [WeiboClient.get_note_info_by_id] $render_data value not found
2026-06-08 20:26:50 MediaCrawler INFO (core.py:218) - [WeiboCrawler.get_note_info_task] Sleeping for 2 seconds after fetching note details https://weibo.com/7798025479/R3bk1ydAR?refer_flag=1001030103_
2026-06-08 20:26:50 MediaCrawler INFO (core.py:238) - [WeiboCrawler.batch_get_notes_comments] note ids:['https://weibo.com/7798025479/R3bk1ydAR?refer_flag=1001030103_']
2026-06-08 20:26:50 MediaCrawler INFO (core.py:255) - [WeiboCrawler.get_note_comments] begin get note_id: https://weibo.com/7798025479/R3bk1ydAR?refer_flag=1001030103_ comments ...
2026-06-08 20:26:52 MediaCrawler INFO (core.py:259) - [WeiboCrawler.get_note_comments] Sleeping for 2 seconds before fetching comments for note https://weibo.com/7798025479/R3bk1ydAR?refer_flag=1001030103_
2026-06-08 20:26:52 MediaCrawler ERROR (client.py:88) - [WeiboClient.request] request GET:https://m.weibo.cn/comments/hotflow?id=https%3A%2F%2Fweibo.com%2F7798025479%2FR3bk1ydAR%3Frefer_flag%3D1001030103_&mid=https%3A%2F%2Fweibo.com%2F7798025479%2FR3bk1ydAR%3Frefer_flag%3D1001030103_&max_id_type=0 err code: 200 res:

<title>微博-出错了</title> <style> html { font-size: 2rem; } 
    @media (max-width: 1024px) {
        html {
            font-size: 1.25rem;
        }
    }

    @media (max-width: 414px) {
        html {
            font-size: 1.06rem;
        }
    }

    @media (max-width: 375px) {
        html {
            font-size: 1rem;
        }
    }
    html,body{
        height:100%;
    }
    body {
        margin: 0;
        padding: 0;
        background-color: #f2f2f2;
    }

    p {
        margin: 0;

    }

    .h5-4box {
        height:100%;
        display: -ms-flexbox;
        display: flex;
        -ms-flex-pack: center;
        justify-content: center;
        -ms-flex-align: center;
        align-items: center;
        -ms-flex-direction: column;
        flex-direction: column;
    }

    .h5-4img {
        /* display: inline-block; */

    }

    .h5-4img img {
        max-width: 100%;
    }

    .h5-4con {
        padding-top: 1.875rem;
        font-size: 0.875rem;
        line-height: 1.2;
        color: #636363;
        text-align: center;
    }

    .h5-5con {
        padding-top: 1.875rem;
        font-size: 0.5rem;
        line-height: 1.2;
        color: #636363;
        text-align: center;
    }

    .btn {
        display: inline-block;
        border: #e86b0f solid 1px;
        margin: 0 0 0 5px;
        padding: 0 10px;
        line-height: 25px;
        font-size: .75rem;
        vertical-align: middle;
        color: #FFF;
        border-radius: 3px;
        background-color: #ff8200;
    }
</style>

出错了，未知错误


返回首页

2026-06-08 20:26:55 MediaCrawler INFO (client.py:148) - [WeiboClient.update_cookies] Cookie updated successfully for ['https://m.weibo.cn'], total: 10 cookies
2026-06-08 20:26:58 MediaCrawler ERROR (client.py:88) - [WeiboClient.request] request GET:https://m.weibo.cn/comments/hotflow?id=https%3A%2F%2Fweibo.com%2F7798025479%2FR3bk1ydAR%3Frefer_flag%3D1001030103_&mid=https%3A%2F%2Fweibo.com%2F7798025479%2FR3bk1ydAR%3Frefer_flag%3D1001030103_&max_id_type=0 err code: 200 res:

<title>微博-出错了</title> <style> html { font-size: 2rem; } 
    @media (max-width: 1024px) {
        html {
            font-size: 1.25rem;
        }
    }

    @media (max-width: 414px) {
        html {
            font-size: 1.06rem;
        }
    }

    @media (max-width: 375px) {
        html {
            font-size: 1rem;
        }
    }
    html,body{
        height:100%;
    }
    body {
        margin: 0;
        padding: 0;
        background-color: #f2f2f2;
    }

    p {
        margin: 0;

    }

    .h5-4box {
        height:100%;
        display: -ms-flexbox;
        display: flex;
        -ms-flex-pack: center;
        justify-content: center;
        -ms-flex-align: center;
        align-items: center;
        -ms-flex-direction: column;
        flex-direction: column;
    }

    .h5-4img {
        /* display: inline-block; */

    }

    .h5-4img img {
        max-width: 100%;
    }

    .h5-4con {
        padding-top: 1.875rem;
        font-size: 0.875rem;
        line-height: 1.2;
        color: #636363;
        text-align: center;
    }

    .h5-5con {
        padding-top: 1.875rem;
        font-size: 0.5rem;
        line-height: 1.2;
        color: #636363;
        text-align: center;
    }

    .btn {
        display: inline-block;
        border: #e86b0f solid 1px;
        margin: 0 0 0 5px;
        padding: 0 10px;
        line-height: 25px;
        font-size: .75rem;
        vertical-align: middle;
        color: #FFF;
        border-radius: 3px;
        background-color: #ff8200;
    }
</style>

出错了，未知错误


返回首页

2026-06-08 20:27:01 MediaCrawler INFO (client.py:148) - [WeiboClient.update_cookies] Cookie updated successfully for ['https://m.weibo.cn'], total: 10 cookies
2026-06-08 20:27:04 MediaCrawler ERROR (client.py:88) - [WeiboClient.request] request GET:https://m.weibo.cn/comments/hotflow?id=https%3A%2F%2Fweibo.com%2F7798025479%2FR3bk1ydAR%3Frefer_flag%3D1001030103_&mid=https%3A%2F%2Fweibo.com%2F7798025479%2FR3bk1ydAR%3Frefer_flag%3D1001030103_&max_id_type=0 err code: 200 res:

<title>微博-出错了</title> <style> html { font-size: 2rem; } 
    @media (max-width: 1024px) {
        html {
            font-size: 1.25rem;
        }
    }

    @media (max-width: 414px) {
        html {
            font-size: 1.06rem;
        }
    }

    @media (max-width: 375px) {
        html {
            font-size: 1rem;
        }
    }
    html,body{
        height:100%;
    }
    body {
        margin: 0;
        padding: 0;
        background-color: #f2f2f2;
    }

    p {
        margin: 0;

    }

    .h5-4box {
        height:100%;
        display: -ms-flexbox;
        display: flex;
        -ms-flex-pack: center;
        justify-content: center;
        -ms-flex-align: center;
        align-items: center;
        -ms-flex-direction: column;
        flex-direction: column;
    }

    .h5-4img {
        /* display: inline-block; */

    }

    .h5-4img img {
        max-width: 100%;
    }

    .h5-4con {
        padding-top: 1.875rem;
        font-size: 0.875rem;
        line-height: 1.2;
        color: #636363;
        text-align: center;
    }

    .h5-5con {
        padding-top: 1.875rem;
        font-size: 0.5rem;
        line-height: 1.2;
        color: #636363;
        text-align: center;
    }

    .btn {
        display: inline-block;
        border: #e86b0f solid 1px;
        margin: 0 0 0 5px;
        padding: 0 10px;
        line-height: 25px;
        font-size: .75rem;
        vertical-align: middle;
        color: #FFF;
        border-radius: 3px;
        background-color: #ff8200;
    }
</style>

出错了，未知错误


返回首页

2026-06-08 20:27:07 MediaCrawler INFO (client.py:148) - [WeiboClient.update_cookies] Cookie updated successfully for ['https://m.weibo.cn'], total: 10 cookies
2026-06-08 20:27:10 MediaCrawler ERROR (client.py:88) - [WeiboClient.request] request GET:https://m.weibo.cn/comments/hotflow?id=https%3A%2F%2Fweibo.com%2F7798025479%2FR3bk1ydAR%3Frefer_flag%3D1001030103_&mid=https%3A%2F%2Fweibo.com%2F7798025479%2FR3bk1ydAR%3Frefer_flag%3D1001030103_&max_id_type=0 err code: 200 res:

<title>微博-出错了</title> <style> html { font-size: 2rem; } 
    @media (max-width: 1024px) {
        html {
            font-size: 1.25rem;
        }
    }

    @media (max-width: 414px) {
        html {
            font-size: 1.06rem;
        }
    }

    @media (max-width: 375px) {
        html {
            font-size: 1rem;
        }
    }
    html,body{
        height:100%;
    }
    body {
        margin: 0;
        padding: 0;
        background-color: #f2f2f2;
    }

    p {
        margin: 0;

    }

    .h5-4box {
        height:100%;
        display: -ms-flexbox;
        display: flex;
        -ms-flex-pack: center;
        justify-content: center;
        -ms-flex-align: center;
        align-items: center;
        -ms-flex-direction: column;
        flex-direction: column;
    }

    .h5-4img {
        /* display: inline-block; */

    }

    .h5-4img img {
        max-width: 100%;
    }

    .h5-4con {
        padding-top: 1.875rem;
        font-size: 0.875rem;
        line-height: 1.2;
        color: #636363;
        text-align: center;
    }

    .h5-5con {
        padding-top: 1.875rem;
        font-size: 0.5rem;
        line-height: 1.2;
        color: #636363;
        text-align: center;
    }

    .btn {
        display: inline-block;
        border: #e86b0f solid 1px;
        margin: 0 0 0 5px;
        padding: 0 10px;
        line-height: 25px;
        font-size: .75rem;
        vertical-align: middle;
        color: #FFF;
        border-radius: 3px;
        background-color: #ff8200;
    }
</style>

出错了，未知错误


返回首页

2026-06-08 20:27:13 MediaCrawler INFO (client.py:148) - [WeiboClient.update_cookies] Cookie updated successfully for ['https://m.weibo.cn'], total: 10 cookies
2026-06-08 20:27:16 MediaCrawler ERROR (client.py:88) - [WeiboClient.request] request GET:https://m.weibo.cn/comments/hotflow?id=https%3A%2F%2Fweibo.com%2F7798025479%2FR3bk1ydAR%3Frefer_flag%3D1001030103_&mid=https%3A%2F%2Fweibo.com%2F7798025479%2FR3bk1ydAR%3Frefer_flag%3D1001030103_&max_id_type=0 err code: 200 res:

<title>微博-出错了</title> <style> html { font-size: 2rem; } 
    @media (max-width: 1024px) {
        html {
            font-size: 1.25rem;
        }
    }

    @media (max-width: 414px) {
        html {
            font-size: 1.06rem;
        }
    }

    @media (max-width: 375px) {
        html {
            font-size: 1rem;
        }
    }
    html,body{
        height:100%;
    }
    body {
        margin: 0;
        padding: 0;
        background-color: #f2f2f2;
    }

    p {
        margin: 0;

    }

    .h5-4box {
        height:100%;
        display: -ms-flexbox;
        display: flex;
        -ms-flex-pack: center;
        justify-content: center;
        -ms-flex-align: center;
        align-items: center;
        -ms-flex-direction: column;
        flex-direction: column;
    }

    .h5-4img {
        /* display: inline-block; */

    }

    .h5-4img img {
        max-width: 100%;
    }

    .h5-4con {
        padding-top: 1.875rem;
        font-size: 0.875rem;
        line-height: 1.2;
        color: #636363;
        text-align: center;
    }

    .h5-5con {
        padding-top: 1.875rem;
        font-size: 0.5rem;
        line-height: 1.2;
        color: #636363;
        text-align: center;
    }

    .btn {
        display: inline-block;
        border: #e86b0f solid 1px;
        margin: 0 0 0 5px;
        padding: 0 10px;
        line-height: 25px;
        font-size: .75rem;
        vertical-align: middle;
        color: #FFF;
        border-radius: 3px;
        background-color: #ff8200;
    }
</style>

出错了，未知错误


返回首页

2026-06-08 20:27:19 MediaCrawler INFO (client.py:148) - [WeiboClient.update_cookies] Cookie updated successfully for ['https://m.weibo.cn'], total: 10 cookies
2026-06-08 20:27:19 MediaCrawler ERROR (core.py:270) - [WeiboCrawler.get_note_comments] may be been blocked, err:RetryError[<Future at 0x2ab09c5a490 state=finished raised DataFetchError>]
2026-06-08 20:27:19 MediaCrawler INFO (core.py:135) - [WeiboCrawler.start] Weibo Crawler finished ...
[Main] Excel files saved successfully
2026-06-08 20:27:19 MediaCrawler INFO (cdp_browser.py:463) - [CDPBrowserManager] Browser connection disconnected
2026-06-08 20:27:19 MediaCrawler INFO (cdp_browser.py:480) - [CDPBrowserManager] Connected to existing browser, skipping process cleanup

📷 错误截图

Image

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with main.py's detail-mode entry point, then inspect WeiboCrawler.get_note_info_task and batch_get_notes_comments in core.py. Trace WeiboClient.get_note_info_by_id and request in client.py using the reported URL and logs. Done means the detail command retrieves the post and comments instead of receiving the Weibo HTML error response.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.