NanmiCoder / NanmiCoder/MediaCrawler

[BUG]

Open
#714 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
65.3k
Forks
12.6k
PR merge metrics
No merged PRs in 30d

Description

🔍 问题检查清单

  • 我已经仔细阅读了项目使用过程中的常见问题汇总
  • 我已经搜索并查看了已关闭的issues
  • 我确认这不是由于滑块验证码、Cookie过期、Cookie提取错误、平台风控等常见原因导致的问题

🐛 问题描述

目前我搜索哔站关键词一切正常,但是搜索贴吧关键词,我设置了headless = false,运行后没有弹出浏览器,然后看到报错如下,我换了1.45和1.55的playwright,并运行了playwright install ,重启了编辑器,挂了或没挂代理都试了。报错依旧。

📝 复现步骤

  1. 基础配置改成搜索 tieba
  2. 运行main文件

💻 运行环境

  • 操作系统: win10
  • Python版本: 3.10.11
  • 是否使用IP代理: kuaidiali
  • 是否使用VPN翻墙软件:无
  • 目标平台(抖音/小红书/微博等): 贴吧

📋 错误日志

2025-09-05 11:08:25 MediaCrawler INFO (core.py:100) - [BaiduTieBaCrawler.search] Begin search baidu tieba keywords
2025-09-05 11:08:25 MediaCrawler INFO (core.py:109) - [BaiduTieBaCrawler.search] Current search keyword: web3
2025-09-05 11:08:25 MediaCrawler INFO (core.py:121) - [BaiduTieBaCrawler.search] search tieba keyword: web3, page: 1
2025-09-05 11:08:26 httpx INFO (_client.py:1740) - HTTP Request: GET https://tieba.baidu.com/f/search/res?isnew=1&qw=web3&rn=10&pn=1&sm=1&only_thread=0
"HTTP/1.1 403 Forbidden"
2025-09-05 11:08:26 MediaCrawler ERROR (client.py:67) - Request failed, method: GET, url: https://tieba.baidu.com/f/search/res?isnew=1&qw=web3&rn=10&pn=1&sm=1&only_thread=0, status code: 403
2025-09-05 11:08:26 MediaCrawler ERROR (client.py:68) - Request failed, response: <meta http-equiv="X-UA-Compatible"content="IE=edge"><meta name="viewport"content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"><title>百度安全验证</title><style>body{margin:0;padding:0}</style><script>function ieOnEnd(script,cb){script.onreadystatechange=function(){if(this.readyState!=='complete'&&this.readyState!=='loaded'){return}this.onreadystatechange=null;cb(null,script)}}function stdOnEnd(script,cb){script.onload=function(){this.onerror=this.onload=null;cb(null,script)};script.onerror=function(e){this.onerror=this.onload=null;cb(new Error(Failed to load ${this.src}),script)}}function loadScript(src,opts,cb){var script=document.createElement('script');if(typeof opts==='function'){cb=opts;opts={}}opts=opts||{};cb=cb||function(){};script.type=opts.type||'text/javascript';script.charset=opts.charset||'utf8';script.async='async'in opts?!!opts.async:true;script.src=src;if(opts.attrs){setAttributes(script,opts.attrs)}if(opts.text){script.text=''+opts.text}var onend='onload'in script?stdOnEnd:ieOnEnd;onend(script,cb);if(!script.onload){stdOnEnd(script,cb)}document.head.appendChild(script)}</script><script>window.BIOC_OPTIONS={ak:'MtwiTOYhbgACaxGXidGoCM3MjRimIPL1',subid:'tb_pc_search_res_acs',ext:'PQoe+IssmaOK3xf4NlXXSz6DLc2zfghSUofs34P/kjc0borMoMeD48KbDyEdRFXvSzkVaZZP7ryCz32EB2pGx7rsKo9QzSvIlf5LJdG9pfU=',backurl:'https://tieba.baidu.com/f/search/res?isnew=1&qw=web3&rn=10&pn=1&sm=1&only_thread=0',closeable:false};</script><script>(function(){var origins=['https://bioc-static.cdn.bcebos.com','https://bioc-static-standby.cdn.bcebos.com','https://seccaptcha.baidu.com/v1/webapi/static','https://seccaptcha-1.baidu.com/v1/webapi/static'];var biocFacadePath='/pages/BiocFacade.js';var bootstrapPath='/pages/bfe_captcha.js';var loadAssetsWithStandby=function(path,cb){var requestOrigins=origins.slice();var doRequest=function(){if(requestOrigins.length===0){cb(new Error('load assets failed'))}else{var origin=requestOrigins.shift();loadScript(origin+path,function(err){if(!err){cb()}else{doRequest()}})}};doRequest()};var loadBioc=function(cb){loadAssetsWithStandby(biocFacadePath,cb)};var loadBootstrap=function(cb){loadAssetsWithStandby(bootstrapPath,cb)};var handlerError=function(err){alert('页面加
载失败,请刷新页面重试');console.error(err)};loadBioc(function(err){if(err){handlerError(err);return}loadBootstrap(function(err){if(err){handlerError(err)}})})})()</script>


## 📷 错误截图
 
<img width="438" height="588" alt="Image" src="https://github.com/user-attachments/assets/08566390-5414-4d00-9ab3-e516eff72178" />

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at BaiduTieBaCrawler.search, identified in core.py around lines 100–121, and trace the request through client.py around lines 67–68. Reproduce the Tieba keyword search on Windows with the reported configuration and inspect the 403 response containing Baidu’s security verification page. Done means the cause and expected handling of this response are established and the Tieba search behavior is verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
playwright, python
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.