NanmiCoder / NanmiCoder/MediaCrawler
在爬取几十条后,浏览器就断开连接,无论时候使用CDP都是这样
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 65.3k
- Forks
- 12.6k
- PR merge metrics
- No merged PRs in 30d
Description
2026-03-30 18:52:43 MediaCrawler INFO (cdp_browser.py:391) - [CDPBrowserManager] Browser connection disconnected
2026-03-30 18:52:43 MediaCrawler INFO (browser_launcher.py:255) - [BrowserLauncher] Closing browser process...
2026-03-30 18:52:43 MediaCrawler INFO (browser_launcher.py:287) - [BrowserLauncher] Browser process closed
Traceback (most recent call last):
File "D:\STUDY-SPACE\Social_media\py\MediaCrawler-main\main.py", line 157, in
run(main, async_cleanup, cleanup_timeout_seconds=15.0, on_first_interrupt=_force_stop)
File "D:\STUDY-SPACE\Social_media\py\MediaCrawler-main\tools\app_runner.py", line 109, in run
asyncio.run(_runner())
File "D:\anaconda\envs\media_crawler\lib\asyncio\runners.py", line 44, in run
return loop.run_until_complete(main)
File "D:\anaconda\envs\media_crawler\lib\asyncio\base_events.py", line 649, in run_until_complete
return future.result()
File "D:\STUDY-SPACE\Social_media\py\MediaCrawler-main\tools\app_runner.py", line 96, in _runner
await app_main()
File "D:\STUDY-SPACE\Social_media\py\MediaCrawler-main\main.py", line 110, in main
await crawler.start()
File "D:\STUDY-SPACE\Social_media\py\MediaCrawler-main\media_platform\xhs\core.py", line 96, in start
await self.search()
File "D:\STUDY-SPACE\Social_media\py\MediaCrawler-main\media_platform\xhs\core.py", line 150, in search
note_details = await asyncio.gather(*task_list)
File "D:\STUDY-SPACE\Social_media\py\MediaCrawler-main\media_platform\xhs\core.py", line 288, in get_note_detail_async_task
raise Exception(f"[get_note_detail_async_task] Failed to get note detail, Id: {note_id}")
Exception: [get_note_detail_async_task] Failed to get note detail, Id: 65dfcfba000000000102b05a
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the XHS search flow from media_platform/xhs/core.py, especially get_note_detail_async_task, using the reported disconnect logs from cdp_browser.py and browser_launcher.py as checkpoints. Trace why the browser connection closes after dozens of notes and verify that the search completes without the reported note-detail exception or premature browser shutdown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devtools
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100