NanmiCoder / NanmiCoder/MediaCrawler

[问题] 贴吧爬虫首次执行任务时 403 Forbidden

Open
#697 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Python
Stars
65.3k
Forks
12.6k
No merged PRs in 30d

Description

⚠️ 提交前确认

  • [√] 我已经仔细阅读了项目使用过程中的常见问题汇总
  • [√] 我已经搜索并查看了已关闭的issues
  • 我确认这不是由于滑块验证码、Cookie过期、Cookie提取错误、平台风控等常见原因导致的问题
    可能是Cookie提取错误、平台风控等常见原因，但是不知道解决方案

❓ 问题描述

贴吧爬虫首次执行任务时 403 Forbidden
登录方式为cookie

🔍 使用场景

  • 目标平台:贴吧
  • 使用功能:用户主页爬取
  • PLATFORM = "tieba" # 平台，xhs | dy | ks | bili | wb | tieba | zhihu
    KEYWORDS = "恐怖推理" # 关键词搜索配置，以英文逗号分隔
    LOGIN_TYPE = "cookie"
    CRAWLER_TYPE = (
    "creator"
    )

💻 环境信息

  • 操作系统: win11
  • Python版本: 3.10
  • 是否使用IP代理: 否
  • 是否使用VPN翻墙软件：否
  • 目标平台(抖音/小红书/微博等):贴吧

📋 错误日志


(.venv) PS C:\Users\xiangzirui\PyCharmMiscProject\MediaCrawler> uv run main.py                                           
warning: `VIRTUAL_ENV=C:\Users\xiangzirui\PyCharmMiscProject\.venv` does not match the project environment path `.venv` and will be ignored; use `--active` to target the active environment instead
2025-08-04 17:00:28 MediaCrawler INFO (core.py:293) - [WeiboCrawler.get_creators_and_notes] Begin get weibo creators
2025-08-04 17:00:28 httpx INFO (_client.py:1740) - HTTP Request: GET https://tieba.baidu.com/home/main?id=tb.1.52671c8a.Jjq52DtBjR5ibxyypGmfxw?t=1582477803&fr=pb "HTTP/1.1 403 Forbidden"
2025-08-04 17:00:28 MediaCrawler ERROR (client.py:67) - Request failed, method: GET, url: https://tieba.baidu.com/home/main?id=tb.1.52671c8a.Jjq52DtBjR5ibxyypGmfxw?t=1582477803&fr=pb, status code: 403
2025-08-04 17:00:28 MediaCrawler ERROR (client.py:68) - Request failed, response: <!DOCTYPE html><html lang="en"><head><meta charset="UTF-8"><meta http-equiv="X-UA-Compatible"content="IE=edge"><meta name="viewport"content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"><title>百
安全验证</title><style>body{margin:0;padding:0}</style></head><body></body><!--loadScript--><script>function ieOnEnd(script,cb){script.onreadystatechange=function(){if(this.readyState!=='complete'&&this.readyState!=='loaded'){return}this.onreadystatechange=null;cb(null,script)}}function stdOnEnd(script,cb b
){script.onload=function(){this.onerror=this.onload=null;cb(null,script)};script.onerror=function(e){this.onerror=this.onload=null;cb(new Error(`Failed to load ${this.src}`),script)}}function loadScript(src,opts,cb){var script=document.createElement('script');if(typeof opts==='function'){cb=opts;opts={}}opt
s=opts||{};cb=cb||function(){};script.type=opts.type||'text/javascript';script.charset=opts.charset||'utf8';script.async='async'in opts?!!opts.async:true;script.src=src;if(opts.attrs){setAttributes(script,opts.attrs)}if(opts.text){script.text=''+opts.text}var onend='onload'in script?stdOnEnd:ieOnEnd;onend(s
cript,cb);if(!script.onload){stdOnEnd(script,cb)}document.head.appendChild(script)}</script><script>window.BIOC_OPTIONS={ak:'MtwiTOYhbgACaxGXidGoCM3MjRimIPL1',subid:'tb_pc_user_home_bfe',ext:'vsGqTDZjAou2DpLyyRZceE1DgSBZ9cLieGztimJtIpOKigW8q4KjeIiFtEY8FOl+I+/MIIjMskDwcs1rKvQBCZ1SW8PoTlz4JSazCFVsLVM=',backur
l:'https://tieba.baidu.com/home/main?id=tb.1.52671c8a.Jjq52DtBjR5ibxyypGmfxw?t=1582477803&fr=pb',closeable:false};</script><script>(function(){var origins=['https://bioc-static.cdn.bcebos.com','https://bioc-static-standby.cdn.bcebos.com','https://seccaptcha.baidu.com/v1/webapi/static','https://seccaptcha-1.
baidu.com/v1/webapi/static'];var biocFacadePath='/pages/BiocFacade.js';var bootstrapPath='/pages/bfe_captcha.js';var loadAssetsWithStandby=function(path,cb){var requestOrigins=origins.slice();var doRequest=function(){if(requestOrigins.length===0){cb(new Error('load assets failed'))}else{var origin=requestOr
igins.shift();loadScript(origin+path,function(err){if(!err){cb()}else{doRequest()}})}};doRequest()};var loadBioc=function(cb){loadAssetsWithStandby(biocFacadePath,cb)};var loadBootstrap=function(cb){loadAssetsWithStandby(bootstrapPath,cb)};var handlerError=function(err){alert('页面加载失败，请刷新页面重试');console.error(err)};loadBioc(function(err){if(err){handlerError(err);return}loadBootstrap(function(err){if(err){handlerError(err)}})})})()</script></html>
2025-08-04 17:00:30 httpx INFO (_client.py:1740) - HTTP Request: GET https://tieba.baidu.com/home/main?id=tb.1.52671c8a.Jjq52DtBjR5ibxyypGmfxw?t=1582477803&fr=pb "HTTP/1.1 403 Forbidden"
2025-08-04 17:00:30 MediaCrawler ERROR (client.py:67) - Request failed, method: GET, url: https://tieba.baidu.com/home/main?id=tb.1.52671c8a.Jjq52DtBjR5ibxyypGmfxw?t=1582477803&fr=pb, status code: 403
2025-08-04 17:00:30 MediaCrawler ERROR (client.py:68) - Request failed, response: <!DOCTYPE html><html lang="en"><head><meta charset="UTF-8"><meta http-equiv="X-UA-Compatible"content="IE=edge"><meta name="viewport"content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"><title>百
安全验证</title><style>body{margin:0;padding:0}</style></head><body></body><!--loadScript--><script>function ieOnEnd(script,cb){script.onreadystatechange=function(){if(this.readyState!=='complete'&&this.readyState!=='loaded'){return}this.onreadystatechange=null;cb(null,script)}}function stdOnEnd(script,cb b
){script.onload=function(){this.onerror=this.onload=null;cb(null,script)};script.onerror=function(e){this.onerror=this.onload=null;cb(new Error(`Failed to load ${this.src}`),script)}}function loadScript(src,opts,cb){var script=document.createElement('script');if(typeof opts==='function'){cb=opts;opts={}}opt
s=opts||{};cb=cb||function(){};script.type=opts.type||'text/javascript';script.charset=opts.charset||'utf8';script.async='async'in opts?!!opts.async:true;script.src=src;if(opts.attrs){setAttributes(script,opts.attrs)}if(opts.text){script.text=''+opts.text}var onend='onload'in script?stdOnEnd:ieOnEnd;onend(s
cript,cb);if(!script.onload){stdOnEnd(script,cb)}document.head.appendChild(script)}</script><script>window.BIOC_OPTIONS={ak:'MtwiTOYhbgACaxGXidGoCM3MjRimIPL1',subid:'tb_pc_user_home_bfe',ext:'6mbd07BnLa08mBTvCZVuuks5FWmWT+68gs99hAdqRsdYOUDN9KXM0GDUst9nzW9PoEblMS3AUnWawo5g9muDzNSJw+KWtxFmmP7SCQA2xeQ=',backur
l:'https://tieba.baidu.com/home/main?id=tb.1.52671c8a.Jjq52DtBjR5ibxyypGmfxw?t=1582477803&fr=pb',closeable:false};</script><script>(function(){var origins=['https://bioc-static.cdn.bcebos.com','https://bioc-static-standby.cdn.bcebos.com','https://seccaptcha.baidu.com/v1/webapi/static','https://seccaptcha-1.
baidu.com/v1/webapi/static'];var biocFacadePath='/pages/BiocFacade.js';var bootstrapPath='/pages/bfe_captcha.js';var loadAssetsWithStandby=function(path,cb){var requestOrigins=origins.slice();var doRequest=function(){if(requestOrigins.length===0){cb(new Error('load assets failed'))}else{var origin=requestOr
igins.shift();loadScript(origin+path,function(err){if(!err){cb()}else{doRequest()}})}};doRequest()};var loadBioc=function(cb){loadAssetsWithStandby(biocFacadePath,cb)};var loadBootstrap=function(cb){loadAssetsWithStandby(bootstrapPath,cb)};var handlerError=function(err){alert('页面加载失败，请刷新页面重试');console.error(err)};loadBioc(function(err){if(err){handlerError(err);return}loadBootstrap(function(err){if(err){handlerError(err)}})})})()</script></html>
2025-08-04 17:00:31 httpx INFO (_client.py:1740) - HTTP Request: GET https://tieba.baidu.com/home/main?id=tb.1.52671c8a.Jjq52DtBjR5ibxyypGmfxw?t=1582477803&fr=pb "HTTP/1.1 403 Forbidden"
2025-08-04 17:00:31 MediaCrawler ERROR (client.py:67) - Request failed, method: GET, url: https://tieba.baidu.com/home/main?id=tb.1.52671c8a.Jjq52DtBjR5ibxyypGmfxw?t=1582477803&fr=pb, status code: 403
2025-08-04 17:00:31 MediaCrawler ERROR (client.py:68) - Request failed, response: <!DOCTYPE html><html lang="en"><head><meta charset="UTF-8"><meta http-equiv="X-UA-Compatible"content="IE=edge"><meta name="viewport"content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"><title>百
安全验证</title><style>body{margin:0;padding:0}</style></head><body></body><!--loadScript--><script>function ieOnEnd(script,cb){script.onreadystatechange=function(){if(this.readyState!=='complete'&&this.readyState!=='loaded'){return}this.onreadystatechange=null;cb(null,script)}}function stdOnEnd(script,cb b
){script.onload=function(){this.onerror=this.onload=null;cb(null,script)};script.onerror=function(e){this.onerror=this.onload=null;cb(new Error(`Failed to load ${this.src}`),script)}}function loadScript(src,opts,cb){var script=document.createElement('script');if(typeof opts==='function'){cb=opts;opts={}}opt
s=opts||{};cb=cb||function(){};script.type=opts.type||'text/javascript';script.charset=opts.charset||'utf8';script.async='async'in opts?!!opts.async:true;script.src=src;if(opts.attrs){setAttributes(script,opts.attrs)}if(opts.text){script.text=''+opts.text}var onend='onload'in script?stdOnEnd:ieOnEnd;onend(s
cript,cb);if(!script.onload){stdOnEnd(script,cb)}document.head.appendChild(script)}</script><script>window.BIOC_OPTIONS={ak:'MtwiTOYhbgACaxGXidGoCM3MjRimIPL1',subid:'tb_pc_user_home_bfe',ext:'PQoe+IssmaOK3xf4NlXXSzFgTDdPVg/S41/uxBp0dZHb/T1Vu+B3Gq7BW4i98n6/tNgg2TZ7RGmo6HSuwfXm0wzZpbV0UAymMGAOAOE+Nz8=',backur
l:'https://tieba.baidu.com/home/main?id=tb.1.52671c8a.Jjq52DtBjR5ibxyypGmfxw?t=1582477803&fr=pb',closeable:false};</script><script>(function(){var origins=['https://bioc-static.cdn.bcebos.com','https://bioc-static-standby.cdn.bcebos.com','https://seccaptcha.baidu.com/v1/webapi/static','https://seccaptcha-1.
baidu.com/v1/webapi/static'];var biocFacadePath='/pages/BiocFacade.js';var bootstrapPath='/pages/bfe_captcha.js';var loadAssetsWithStandby=function(path,cb){var requestOrigins=origins.slice();var doRequest=function(){if(requestOrigins.length===0){cb(new Error('load assets failed'))}else{var origin=requestOr
igins.shift();loadScript(origin+path,function(err){if(!err){cb()}else{doRequest()}})}};doRequest()};var loadBioc=function(cb){loadAssetsWithStandby(biocFacadePath,cb)};var loadBootstrap=function(cb){loadAssetsWithStandby(bootstrapPath,cb)};var handlerError=function(err){alert('页面加载失败，请刷新页面重试');console.error(err)};loadBioc(function(err){if(err){handlerError(err);return}loadBootstrap(function(err){if(err){handlerError(err)}})})})()</script></html>
Traceback (most recent call last):
  File "C:\Users\xiangzirui\PyCharmMiscProject\MediaCrawler\.venv\lib\site-packages\tenacity\_asyncio.py", line 50, in __call__
    result = await fn(*args, **kwargs)
  File "C:\Users\xiangzirui\PyCharmMiscProject\MediaCrawler\media_platform\tieba\client.py", line 69, in request
    raise Exception(f"Request failed, method: {method}, url: {url}, status code: {response.status_code}")
Exception: Request failed, method: GET, url: https://tieba.baidu.com/home/main?id=tb.1.52671c8a.Jjq52DtBjR5ibxyypGmfxw?t=1582477803&fr=pb, status code: 403

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\xiangzirui\PyCharmMiscProject\MediaCrawler\main.py", line 78, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File "C:\Users\xiangzirui\AppData\Roaming\uv\python\cpython-3.9.23-windows-x86_64-none\lib\asyncio\base_events.py", line 647, in run_until_complete
    return future.result()
  File "C:\Users\xiangzirui\PyCharmMiscProject\MediaCrawler\main.py", line 65, in main
    await crawler.start()
  File "C:\Users\xiangzirui\PyCharmMiscProject\MediaCrawler\media_platform\tieba\core.py", line 88, in start
    await self.get_creators_and_notes()
  File "C:\Users\xiangzirui\PyCharmMiscProject\MediaCrawler\media_platform\tieba\core.py", line 297, in get_creators_and_notes
    creator_page_html_content = await self.tieba_client.get_creator_info_by_url(
  File "C:\Users\xiangzirui\PyCharmMiscProject\MediaCrawler\media_platform\tieba\client.py", line 310, in get_creator_info_by_url
    page_content = await self.request(method="GET", url=creator_url, return_ori_content=True)
  File "C:\Users\xiangzirui\PyCharmMiscProject\MediaCrawler\.venv\lib\site-packages\tenacity\_asyncio.py", line 88, in async_wrapped
    return await fn(*args, **kwargs)
  File "C:\Users\xiangzirui\PyCharmMiscProject\MediaCrawler\.venv\lib\site-packages\tenacity\_asyncio.py", line 47, in __call__
    do = self.iter(retry_state=retry_state)
  File "C:\Users\xiangzirui\PyCharmMiscProject\MediaCrawler\.venv\lib\site-packages\tenacity\__init__.py", line 326, in iter
    raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x29e75083910 state=finished raised Exception>]

📷 错误截图

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the traceback path in media_platform/tieba/core.py and media_platform/tieba/client.py, then inspect the request made by get_creator_info_by_url for the Tieba creator page. Reproduce the 403 response using the stated cookie-login configuration and compare it with the returned security-verification page. Done means the cause and a verified resolution or limitation are clearly established.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.