NanmiCoder / NanmiCoder/MediaCrawler

爬小红书出现频繁访问的错误

Open
#447 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
65.3k
Forks
12.6k
PR merge metrics
No merged PRs in 30d

Description

Traceback (most recent call last):
File "D:\pythonProject.venv\MediaCrawler-main\Lib\site-packages\tenacity_asyncio.py", line 50, in call
result = await fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\clpq\MediaCrawler-main\media_platform\xhs\client.py", line 99, in request
raise DataFetchError(data.get("msg", None))
media_platform.xhs.exception.DataFetchError: 访问频次异常,请勿频繁操作或重启试试

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "D:\clpq\MediaCrawler-main\main.py", line 55, in
asyncio.get_event_loop().run_until_complete(main())
File "C:\Users\zxnb\AppData\Local\Programs\Python\Python312\Lib\asyncio\base_events.py", line 687, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "D:\clpq\MediaCrawler-main\main.py", line 45, in main
await crawler.start()
File "D:\clpq\MediaCrawler-main\media_platform\xhs\core.py", line 78, in start
await self.search()
File "D:\clpq\MediaCrawler-main\media_platform\xhs\core.py", line 138, in search
await self.batch_get_note_comments(note_id_list)
File "D:\clpq\MediaCrawler-main\media_platform\xhs\core.py", line 252, in batch_get_note_comments
await asyncio.gather(*task_list)
File "D:\clpq\MediaCrawler-main\media_platform\xhs\core.py", line 258, in get_comments
await self.xhs_client.get_note_all_comments(
File "D:\clpq\MediaCrawler-main\media_platform\xhs\client.py", line 288, in get_note_all_comments
comments_res = await self.get_note_comments(note_id, comments_cursor)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\clpq\MediaCrawler-main\media_platform\xhs\client.py", line 249, in get_note_comments
return await self.get(uri, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\clpq\MediaCrawler-main\media_platform\xhs\client.py", line 116, in get
return await self.request(method="GET", url=f"{self.host}{final_uri}", headers=headers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\pythonProject.venv\MediaCrawler-main\Lib\site-packages\tenacity_asyncio.py", line 88, in async_wrapped
return await fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\pythonProject.venv\MediaCrawler-main\Lib\site-packages\tenacity_asyncio.py", line 47, in call
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\pythonProject.venv\MediaCrawler-main\Lib\site-packages\tenacity_init
.py", line 326, in iter
raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x24f2e98f8c0 state=finished raised DataFetchError>]

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with media_platform/xhs/client.py, especially request, get_note_comments, and get, then trace the call from media_platform/xhs/core.py get_comments. Reproduce the frequent-access error shown in the traceback and determine the intended handling; the issue does not specify a fix or completion condition.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.