NanmiCoder / NanmiCoder/MediaCrawler

[问题] IP代理配置后无法使用

Open
#647 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Python
Stars
65.3k
Forks
12.6k
PR merge metrics
No merged PRs in 30d

Description

⚠️ 提交前确认

  • 我已经仔细阅读了项目使用过程中的常见问题汇总
  • 我已经搜索并查看了已关闭的issues
  • 我确认这不是由于滑块验证码、Cookie过期、Cookie提取错误、平台风控等常见原因导致的问题

❓ 问题描述

🔍 使用场景

  • 目标平台: (如:小红书/抖音/微博等)
  • 使用功能: (如:关键词搜索/用户主页爬取等)

💻 环境信息

  • 操作系统:
  • Python版本: 2.10
  • 是否使用IP代理: 是
  • 是否使用VPN翻墙软件:是
  • 目标平台(抖音/小红书/微博等): 小红书

📋 错误日志

2025-07-11 16:07:50 MediaCrawler INFO (proxy_ip_pool.py:59) - [ProxyIpPool._is_valid_proxy] testing 58.19.55.17 is it valid 
2025-07-11 16:07:55 MediaCrawler INFO (core.py:71) - [XiaoHongShuCrawler] 使用标准模式启动浏览器
2025-07-11 16:07:55 MediaCrawler INFO (core.py:440) - [XiaoHongShuCrawler.launch_browser] Begin create browser context ...
Traceback (most recent call last):
  File "D:\MediaCrawler-main\main.py", line 67, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File "D:\Anaconda_envs\envs\cralwer\lib\asyncio\base_events.py", line 649, in run_until_complete
    return future.result()
  File "D:\MediaCrawler-main\main.py", line 57, in main
    await crawler.start()
  File "D:\MediaCrawler-main\media_platform\xhs\core.py", line 91, in start
    await self.context_page.goto(self.index_url)
  File "D:\Anaconda_envs\envs\cralwer\lib\site-packages\playwright\async_api\_generated.py", line 8818, in goto
    await self._impl_obj.goto(
  File "D:\Anaconda_envs\envs\cralwer\lib\site-packages\playwright\_impl\_page.py", line 524, in goto
    return await self._main_frame.goto(**locals_to_params(locals()))
  File "D:\Anaconda_envs\envs\cralwer\lib\site-packages\playwright\_impl\_frame.py", line 145, in goto
    await self._channel.send("goto", locals_to_params(locals()))
  File "D:\Anaconda_envs\envs\cralwer\lib\site-packages\playwright\_impl\_connection.py", line 59, in send
    return await self._connection.wrap_api_call(
  File "D:\Anaconda_envs\envs\cralwer\lib\site-packages\playwright\_impl\_connection.py", line 514, in wrap_api_call
    raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None
playwright._impl._errors.Error: Page.goto: net::ERR_PROXY_CONNECTION_FAILED at https://www.xiaohongshu.com/
Call log:
navigating to "https://www.xiaohongshu.com/", waiting until "load"

📷 错误截图

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with media_platform/xhs/core.py and proxy_ip_pool.py, then reproduce the command shown in the report and inspect the Playwright Page.goto failure. Verify how the proxy is validated and passed to the browser context; done means the Xiaohongshu page loads successfully with the configured proxy and the failure is covered by a clear diagnostic or test.

Written by the indexing model from the issue text.

Assessment

Tech stack
playwright, python
Domain
backend, networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.