NanmiCoder / NanmiCoder/MediaCrawler

小红书用户主页爬取是否支持从上一次中断的地方继续爬取?

Open
#544 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Python
Stars
65.3k
Forks
12.6k
PR merge metrics
No merged PRs in 30d

Description

⚠️ 提交前确认

  • [☑️] 我已经仔细阅读了项目使用过程中的常见问题汇总
  • [☑️] 我已经搜索并查看了已关闭的issues
  • [☑️] 我确认这不是由于滑块验证码、Cookie过期、Cookie提取错误、平台风控等常见原因导致的问题

❓ 问题描述

python main.py --platform xhs --lt cookie --type creator,爬取小红书用户主页的所有帖子,爬取300条帖子后,提示”访问频次异常,请勿频繁操作或重启试试“。
第二次运行时,发现是从头开始爬取,重复的帖子信息写入json文件。爬取300条帖子后,再次提示”访问频次异常,请勿频繁操作或重启试试“。
尝试修改base_config.py中的“START_PAGE = 11”,没有用。
请问用户主页爬取是否支持从上一次中断的地方继续爬取?后续可否增加这个功能?d3eeccb好像只针对搜索的情况。

🔍 使用场景

  • 目标平台: 小红书
  • 使用功能: 用户主页爬取

💻 环境信息

  • 操作系统: MacOS 15.2
  • Python版本: Python 3.9.20
  • 是否使用IP代理: 否
  • 是否使用VPN翻墙软件:否
  • 目标平台(抖音/小红书/微博等): 小红书

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the python main.py --platform xhs --lt cookie --type creator entry point and inspect base_config.py, especially START_PAGE, alongside commit d3eeccb. Compare the existing search-resume behavior with creator-page crawling; done means an interrupted crawl resumes from the appropriate position without rewriting duplicate posts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.