NanmiCoder / NanmiCoder/MediaCrawler

[BUG] 贴吧指定贴吧ID列表，然后爬取一级评论和二级评论时，获取不到评论。

Open
#742 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
65.3k
Forks
12.6k
No merged PRs in 30d

Description

🔍 问题检查清单

  • 我已经仔细阅读了项目使用过程中的常见问题汇总
  • 我已经搜索并查看了已关闭的issues
  • 我确认这不是由于滑块验证码、Cookie过期、Cookie提取错误、平台风控等常见原因导致的问题

🐛 问题描述

贴吧指定贴吧ID列表，然后爬取一级评论和二级评论时，获取不到评论的数据。

📝 复现步骤

💻 运行环境

  • 操作系统: conda
  • Python版本: 3.11
  • 是否使用IP代理: 否
  • 是否使用VPN翻墙软件：否
  • 目标平台(抖音/小红书/微博等): 贴吧

📋 错误日志

(base) D:\PyCharm\MediaCrawler-main>uv run main.py --platform tieba --lt qrcode --type detail
2025-10-20 16:50:04 MediaCrawler INFO (core.py:75) - [BaiduTieBaCrawler] 使用CDP模式启动浏览器
2025-10-20 16:50:09 MediaCrawler INFO (cdp_browser.py:94) - [CDPBrowserManager] 检测到浏览器: Google Chrome (Unknown Version)
2025-10-20 16:50:09 MediaCrawler INFO (cdp_browser.py:97) - [CDPBrowserManager] 浏览器路径: C:\Program Files\Google\Chrome\Application\chrome.exe
2025-10-20 16:50:09 MediaCrawler INFO (cdp_browser.py:137) - [CDPBrowserManager] 用户数据目录: D:\PyCharm\MediaCrawler-main\browser_data\cdp_tieba_user_data_dir
2025-10-20 16:50:09 MediaCrawler INFO (browser_launcher.py:154) - [BrowserLauncher] 启动浏览器: C:\Program Files\Google\Chrome\Application\chrome.exe
2025-10-20 16:50:09 MediaCrawler INFO (browser_launcher.py:155) - [BrowserLauncher] 调试端口: 9222
2025-10-20 16:50:09 MediaCrawler INFO (browser_launcher.py:156) - [BrowserLauncher] 无头模式: False
2025-10-20 16:50:09 MediaCrawler INFO (browser_launcher.py:186) - [BrowserLauncher] 等待浏览器在端口 9222 上准备就绪...
2025-10-20 16:50:09 MediaCrawler INFO (browser_launcher.py:195) - [BrowserLauncher] 浏览器已在端口 9222 上准备就绪
2025-10-20 16:50:10 MediaCrawler INFO (cdp_browser.py:111) - [CDPBrowserManager] CDP端口 9222 可访问
2025-10-20 16:50:10 httpx INFO (_client.py:1740) - HTTP Request: GET http://localhost:9222/json/version "HTTP/1.1 200 OK"
2025-10-20 16:50:10 MediaCrawler INFO (cdp_browser.py:175) - [CDPBrowserManager] 获取到浏览器WebSocket URL: ws://localhost:9222/devtools/browser/696748be-b579-4df2-a438-41fc9c794aef
2025-10-20 16:50:10 MediaCrawler INFO (cdp_browser.py:194) - [CDPBrowserManager] 正在通过CDP连接到浏览器: ws://localhost:9222/devtools/browser/696748be-b579-4df2-a438-41fc9c794aef
2025-10-20 16:50:11 MediaCrawler INFO (cdp_browser.py:200) - [CDPBrowserManager] 成功连接到浏览器
2025-10-20 16:50:11 MediaCrawler INFO (cdp_browser.py:201) - [CDPBrowserManager] 浏览器上下文数量: 1
2025-10-20 16:50:11 MediaCrawler INFO (cdp_browser.py:226) - [CDPBrowserManager] 使用现有的浏览器上下文
2025-10-20 16:50:11 MediaCrawler INFO (core.py:647) - [TieBaCrawler] CDP浏览器信息: {'version': '116.0.5845.180', 'contexts_count': 1, 'debug_port': 9222, 'is_connected': True}
2025-10-20 16:50:11 MediaCrawler INFO (core.py:485) - [TieBaCrawler] Injecting anti-detection scripts...
2025-10-20 16:50:11 MediaCrawler INFO (core.py:534) - [TieBaCrawler] Anti-detection scripts injected
2025-10-20 16:50:11 MediaCrawler INFO (core.py:401) - [TieBaCrawler] 模拟真实用户访问路径...
2025-10-20 16:50:11 MediaCrawler INFO (core.py:405) - [TieBaCrawler] Step 1: 访问百度首页 https://www.baidu.com/
2025-10-20 16:50:11 MediaCrawler INFO (core.py:409) - [TieBaCrawler] Step 2: 等待 2秒 模拟用户浏览...
2025-10-20 16:50:13 MediaCrawler INFO (core.py:413) - [TieBaCrawler] Step 3: 查找并点击'贴吧'链接...
2025-10-20 16:50:13 MediaCrawler INFO (core.py:428) - [TieBaCrawler] 找到贴吧链接 (selector: a[href="http://tieba.baidu.com/"])
2025-10-20 16:50:13 MediaCrawler INFO (core.py:439) - [TieBaCrawler] Step 4: 点击贴吧链接...
2025-10-20 16:50:13 MediaCrawler INFO (core.py:443) - [TieBaCrawler] 链接target属性: _blank
2025-10-20 16:50:13 MediaCrawler INFO (core.py:447) - [TieBaCrawler] 链接会在新标签页打开,等待新页面...
2025-10-20 16:50:15 MediaCrawler INFO (core.py:461) - [TieBaCrawler] ✅ 已切换到新标签页 (贴吧页面)
2025-10-20 16:50:15 MediaCrawler INFO (core.py:469) - [TieBaCrawler] Step 5: 页面加载完成,等待 2秒...
2025-10-20 16:50:17 MediaCrawler INFO (core.py:473) - [TieBaCrawler] ✅ 成功通过百度首页进入贴吧! 当前URL: https://tieba.baidu.com/
2025-10-20 16:50:17 MediaCrawler INFO (core.py:548) - [TieBaCrawler.create_tieba_client] Begin create tieba API client...
2025-10-20 16:50:17 MediaCrawler INFO (core.py:552) - [TieBaCrawler.create_tieba_client] Extracted User-Agent from browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36
2025-10-20 16:50:17 MediaCrawler INFO (client.py:174) - [BaiduTieBaClient.pong] Begin to check tieba login state by cookies...
2025-10-20 16:50:17 MediaCrawler INFO (client.py:190) - [BaiduTieBaClient.pong] Login state verified by cookies (STOKEN: True, PTOKEN: True, BDUSS: True)
2025-10-20 16:50:17 MediaCrawler INFO (core.py:272) - [BaiduTieBaCrawler.get_note_detail] Begin get note detail, note_id: 10125670400
2025-10-20 16:50:17 MediaCrawler INFO (client.py:287) - [BaiduTieBaClient.get_note_by_id] 访问帖子详情页面: https://tieba.baidu.com/p/10125670400
2025-10-20 16:50:20 MediaCrawler INFO (client.py:298) - [BaiduTieBaClient.get_note_by_id] 成功获取帖子详情HTML,长度: 443217
2025-10-20 16:50:22 MediaCrawler INFO (core.py:279) - [TieBaCrawler.get_note_detail_async_task] Sleeping for 2 seconds after fetching note details 10125670400
2025-10-20 16:50:22 MediaCrawler INFO (core.py:272) - [BaiduTieBaCrawler.get_note_detail] Begin get note detail, note_id: 9922347749
2025-10-20 16:50:22 MediaCrawler INFO (client.py:287) - [BaiduTieBaClient.get_note_by_id] 访问帖子详情页面: https://tieba.baidu.com/p/9922347749
2025-10-20 16:50:27 MediaCrawler INFO (client.py:298) - [BaiduTieBaClient.get_note_by_id] 成功获取帖子详情HTML,长度: 311936
2025-10-20 16:50:29 MediaCrawler INFO (core.py:279) - [TieBaCrawler.get_note_detail_async_task] Sleeping for 2 seconds after fetching note details 9922347749
2025-10-20 16:50:29 MediaCrawler INFO (__init__.py:65) - [store.tieba.update_tieba_note] tieba note: {'note_id': '10125670400', 'title': '武汉大学回应杨某某论文，15岁初中生怎么看？_武汉大学吧_百度贴吧', 'desc': '武汉大学回应杨某某论..我 觉得我们可以注意到一个细节，就是武汉大学在回应杨某某论文的时候，用了“百余处错误”这个词儿，你想如果武汉大学单纯是为了保她，说“多处错误”不就行了吗？为什么一定要用“百余处错误”（敲黑板）这个词儿呢', 'note_url': 'https://tieba.baidu.com/p/10125670400', 'publish_time': '', 'user_link': 'https://tieba.baidu.com/home/main?id=tb.1.50941e3f.I-cgiqszehk0GUQwl5Kmbg&fr=pb&ie=utf-8', 'user_nickname': '峰dH', 'user_avatar': '//himg.bdimg.com/sys/portrait/item/tb.1.50941e3f.I-cgiqszehk0GUQwl5Kmbg', 'tieba_name': '武汉大学吧', 'tieba_link': 'https://tieba.baidu.com/f?kw=%E6%AD%A6%E6%B1%89%E5%A4%A7%E5%AD%A6&ie=utf-8', 'total_replay_num': 131, 'total_replay_page': 1, 'ip_location': '', 'source_keyword': '', 'last_modify_ts': 1760950229319}
2025-10-20 16:50:29 MediaCrawler INFO (__init__.py:65) - [store.tieba.update_tieba_note] tieba note: {'note_id': '9922347749', 'title': '武大学生都干了吗', 'desc': '武大学生都干了吗..首先明确的是杨就是个cs，学术不端品行恶劣，败诉了还要对男方穷追猛打的cs，也希望学校尽快拿出一个态度出来，本人也是在直通车上反映了这件事，但是本科生人微言轻能做的实在有限。鼠鼠只是一个最最最普通的学', 'note_url': 'https://tieba.baidu.com/p/9922347749', 'publish_time': '', 'user_link': 'https://tieba.baidu.com/home/main?id=tb.1.1b45acdf.VKojzXhcYk2N_c-Cp6opyw&fr=pb&ie=utf-8', 'user_nickname': 'szczs', 'user_avatar': '//himg.bdimg.com/sys/portrait/item/tb.1.1b45acdf.VKojzXhcYk2N_c-Cp6opyw', 'tieba_name': '武汉大学吧', 'tieba_link': 'https://tieba.baidu.com/f?kw=%E6%AD%A6%E6%B1%89%E5%A4%A7%E5%AD%A6&ie=utf-8', 'total_replay_num': 19, 'total_replay_page': 1, 'ip_location': '', 'source_keyword': '', 'last_modify_ts': 1760950229324}
2025-10-20 16:50:29 MediaCrawler INFO (core.py:333) - [BaiduTieBaCrawler.get_comments] Begin get note id comments 10125670400
2025-10-20 16:50:31 MediaCrawler INFO (core.py:339) - [TieBaCrawler.get_comments_async_task] Sleeping for 2 seconds before fetching comments for note 10125670400
2025-10-20 16:50:31 MediaCrawler INFO (client.py:335) - [BaiduTieBaClient.get_note_all_comments] 访问评论页面: https://tieba.baidu.com/p/10125670400?pn=1
2025-10-20 16:50:35 MediaCrawler INFO (client.py:353) - [BaiduTieBaClient.get_note_all_comments] 第1页没有评论,停止爬取
2025-10-20 16:50:35 MediaCrawler INFO (client.py:377) - [BaiduTieBaClient.get_note_all_comments] 共获取 0 条一级评论
2025-10-20 16:50:35 MediaCrawler INFO (core.py:333) - [BaiduTieBaCrawler.get_comments] Begin get note id comments 9922347749
2025-10-20 16:50:37 MediaCrawler INFO (core.py:339) - [TieBaCrawler.get_comments_async_task] Sleeping for 2 seconds before fetching comments for note 9922347749
2025-10-20 16:50:37 MediaCrawler INFO (client.py:335) - [BaiduTieBaClient.get_note_all_comments] 访问评论页面: https://tieba.baidu.com/p/9922347749?pn=1
2025-10-20 16:50:40 MediaCrawler INFO (client.py:353) - [BaiduTieBaClient.get_note_all_comments] 第1页没有评论,停止爬取
2025-10-20 16:50:40 MediaCrawler INFO (client.py:377) - [BaiduTieBaClient.get_note_all_comments] 共获取 0 条一级评论
2025-10-20 16:50:40 MediaCrawler INFO (core.py:133) - [BaiduTieBaCrawler.start] Tieba Crawler finished ...
在此粘贴错误日志

📷 错误截图

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in core.py at TieBaCrawler.get_comments and client.py at BaiduTieBaClient.get_note_all_comments. Reproduce with the shown uv run main.py --platform tieba --lt qrcode --type detail command and inspect why the fetched Tieba pages yield no comments. Done means the specified Tieba posts produce first-level and second-level comment data instead of stopping at zero comments.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.