NanmiCoder / NanmiCoder/MediaCrawler

在爬取几十条后,浏览器就断开连接,无论时候使用CDP都是这样

Open
#857 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug question
Dominant language
Python
Stars
65.3k
Forks
12.6k
PR merge metrics
No merged PRs in 30d

Description

2026-03-30 18:52:43 MediaCrawler INFO (cdp_browser.py:391) - [CDPBrowserManager] Browser connection disconnected
2026-03-30 18:52:43 MediaCrawler INFO (browser_launcher.py:255) - [BrowserLauncher] Closing browser process...
2026-03-30 18:52:43 MediaCrawler INFO (browser_launcher.py:287) - [BrowserLauncher] Browser process closed
Traceback (most recent call last):
File "D:\STUDY-SPACE\Social_media\py\MediaCrawler-main\main.py", line 157, in
run(main, async_cleanup, cleanup_timeout_seconds=15.0, on_first_interrupt=_force_stop)
File "D:\STUDY-SPACE\Social_media\py\MediaCrawler-main\tools\app_runner.py", line 109, in run
asyncio.run(_runner())
File "D:\anaconda\envs\media_crawler\lib\asyncio\runners.py", line 44, in run
return loop.run_until_complete(main)
File "D:\anaconda\envs\media_crawler\lib\asyncio\base_events.py", line 649, in run_until_complete
return future.result()
File "D:\STUDY-SPACE\Social_media\py\MediaCrawler-main\tools\app_runner.py", line 96, in _runner
await app_main()
File "D:\STUDY-SPACE\Social_media\py\MediaCrawler-main\main.py", line 110, in main
await crawler.start()
File "D:\STUDY-SPACE\Social_media\py\MediaCrawler-main\media_platform\xhs\core.py", line 96, in start
await self.search()
File "D:\STUDY-SPACE\Social_media\py\MediaCrawler-main\media_platform\xhs\core.py", line 150, in search
note_details = await asyncio.gather(*task_list)
File "D:\STUDY-SPACE\Social_media\py\MediaCrawler-main\media_platform\xhs\core.py", line 288, in get_note_detail_async_task
raise Exception(f"[get_note_detail_async_task] Failed to get note detail, Id: {note_id}")
Exception: [get_note_detail_async_task] Failed to get note detail, Id: 65dfcfba000000000102b05a

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the XHS search flow from media_platform/xhs/core.py, especially get_note_detail_async_task, using the reported disconnect logs from cdp_browser.py and browser_launcher.py as checkpoints. Trace why the browser connection closes after dozens of notes and verify that the search completes without the reported note-detail exception or premature browser shutdown.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.