NanmiCoder / NanmiCoder/MediaCrawler
抖音get_aweme_detail显示request params incrr,但是视频的评论成功抓取了。
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 65.3k
- Forks
- 12.6k
- PR merge metrics
- No merged PRs in 30d
Description
抖音的creator模式登录完成后直接显示account block崩出来,停了半个多月没抓,更新项目再跑还是blocked,死活找不到问题。
突发奇想试试detail模式,发现错误只出在DouYinCrawler.get_aweme_detail,抓评论的DouYinCrawler.get_comments是成功的。
非常迷茫毫无思路。
`
(.venv) kayzh@debianLocal:~/MediaCrawler$ python main.py --platform dy --lt qrcode --type detail
2024-10-31 18:10:58 MediaCrawler INFO (db.py:73) - [init_db] start init mediacrawler db connect object
2024-10-31 18:10:58 MediaCrawler INFO (db.py:75) - [init_db] end init mediacrawler db connect object
2024-10-31 18:10:58 MediaCrawler INFO (proxy_ip_pool.py:59) - [ProxyIpPool._is_valid_proxy] testing 36.151.192.212 is it valid
2024-10-31 18:11:10 MediaCrawler INFO (login.py:116) - [DouYinLogin.login_by_qrcode] Begin login douyin by qrcode...
(eog:55849): Handy-WARNING **: 18:11:11.171: Using GtkSettings:gtk-application-prefer-dark-theme together with HdyStyleManager is unsupported. Please use HdyStyleManager:color-scheme instead.
2024-10-31 18:11:19 MediaCrawler INFO (login.py:70) - [DouYinLogin.begin] login finished then check login state ...
2024-10-31 18:11:40 MediaCrawler INFO (login.py:79) - [DouYinLogin.begin] Login successful then wait for 5 seconds redirect ...
2024-10-31 18:11:45 MediaCrawler ERROR (client.py:102) - request params incrr, response.text:
2024-10-31 18:11:45 MediaCrawler ERROR (core.py:150) - [DouYinCrawler.get_aweme_detail] Get aweme detail error: account blocked,
2024-10-31 18:11:46 MediaCrawler ERROR (client.py:102) - request params incrr, response.text:
2024-10-31 18:11:46 MediaCrawler ERROR (core.py:150) - [DouYinCrawler.get_aweme_detail] Get aweme detail error: account blocked,
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7303962837465432873, content: 用python练模型和测试啊,但部署肯定要用c++重写呀,不矛盾啊
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7281145506382168892, content: python的效率是我见过最低的,还有比python更低的吗
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7304186472210744104, content: C++的基础库真是少的可怜,有一次我想读取个csv文件,我不假思索写了个line.split(","),结果你猜怎么着,报错了[流泪][流泪][流泪]
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7308621761716781843, content: python 开发效率高,运行效率低。C++开发效率低,运行效率高。前者可以用高算力机器弥补,后者可以用高水平人力弥补。
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7281072117196522281, content: 咱又不干大数据商业化,python够用了呗
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7281079667602883382, content: 说白点,晚出现的语言底层都是c,c++,函数,套件齐全就是慢。c++不全要自己写,难得是写代码的时间。
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7314371957528527626, content: Python就是c++基础库的精简版
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7313937205062763291, content: 简单好用,工具多,社区环境好
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7313453315387360009, content: C是打工人,Python是老板
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7282193842115347234, content: 建议取消所有语言,只留c,再难学也比一年换一个强
2024-10-31 18:11:47 MediaCrawler INFO (core.py:185) - [DouYinCrawler.get_comments] aweme_id: 7280854932641664319 comments have all been obtained and filtered ...
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7431204786026791739, content: 求源代码
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7430467175989609268, content: 求源代码
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7413519452623192868, content: 只想学游戏脚本该怎么学python
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7316485654572008202, content: 收费吗
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7298589399850156850, content: 求源码
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7300272165453415222, content: 求源码[鼓掌][鼓掌][鼓掌]
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7319455921330897664, content: 求源码
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7426388605792174882, content: 求源代码
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7310162488493900579, content: 求源代码
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7281223217397580559, content: 求源代码[流泪]
2024-10-31 18:11:49 MediaCrawler INFO (core.py:185) - [DouYinCrawler.get_comments] aweme_id: 7202432992642387233 comments have all been obtained and filtered ...
2024-10-31 18:11:49 MediaCrawler INFO (core.py:85) - [DouYinCrawler.start] Douyin Crawler finished ...
2024-10-31 18:11:49 MediaCrawler INFO (db.py:84) - [close] close mediacrawler db pool
`
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the reported command with python main.py --platform dy --lt qrcode --type detail, then inspect client.py, core.py, and the DouYinCrawler.get_aweme_detail and get_comments entry points. Compare their request results and logs; done means the detail request no longer reports the described account-blocked failure while comment retrieval remains functional.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100