NanmiCoder / NanmiCoder/MediaCrawler

抖音get_aweme_detail显示request params incrr,但是视频的评论成功抓取了。

Open
#475 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

待确认
Dominant language
Python
Stars
65.3k
Forks
12.6k
PR merge metrics
No merged PRs in 30d

Description

抖音的creator模式登录完成后直接显示account block崩出来,停了半个多月没抓,更新项目再跑还是blocked,死活找不到问题。

突发奇想试试detail模式,发现错误只出在DouYinCrawler.get_aweme_detail,抓评论的DouYinCrawler.get_comments是成功的。

非常迷茫毫无思路。

`
(.venv) kayzh@debianLocal:~/MediaCrawler$ python main.py --platform dy --lt qrcode --type detail
2024-10-31 18:10:58 MediaCrawler INFO (db.py:73) - [init_db] start init mediacrawler db connect object
2024-10-31 18:10:58 MediaCrawler INFO (db.py:75) - [init_db] end init mediacrawler db connect object
2024-10-31 18:10:58 MediaCrawler INFO (proxy_ip_pool.py:59) - [ProxyIpPool._is_valid_proxy] testing 36.151.192.212 is it valid
2024-10-31 18:11:10 MediaCrawler INFO (login.py:116) - [DouYinLogin.login_by_qrcode] Begin login douyin by qrcode...

(eog:55849): Handy-WARNING **: 18:11:11.171: Using GtkSettings:gtk-application-prefer-dark-theme together with HdyStyleManager is unsupported. Please use HdyStyleManager:color-scheme instead.
2024-10-31 18:11:19 MediaCrawler INFO (login.py:70) - [DouYinLogin.begin] login finished then check login state ...
2024-10-31 18:11:40 MediaCrawler INFO (login.py:79) - [DouYinLogin.begin] Login successful then wait for 5 seconds redirect ...
2024-10-31 18:11:45 MediaCrawler ERROR (client.py:102) - request params incrr, response.text:
2024-10-31 18:11:45 MediaCrawler ERROR (core.py:150) - [DouYinCrawler.get_aweme_detail] Get aweme detail error: account blocked,
2024-10-31 18:11:46 MediaCrawler ERROR (client.py:102) - request params incrr, response.text:
2024-10-31 18:11:46 MediaCrawler ERROR (core.py:150) - [DouYinCrawler.get_aweme_detail] Get aweme detail error: account blocked,
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7303962837465432873, content: 用python练模型和测试啊,但部署肯定要用c++重写呀,不矛盾啊
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7281145506382168892, content: python的效率是我见过最低的,还有比python更低的吗
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7304186472210744104, content: C++的基础库真是少的可怜,有一次我想读取个csv文件,我不假思索写了个line.split(","),结果你猜怎么着,报错了[流泪][流泪][流泪]
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7308621761716781843, content: python 开发效率高,运行效率低。C++开发效率低,运行效率高。前者可以用高算力机器弥补,后者可以用高水平人力弥补。
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7281072117196522281, content: 咱又不干大数据商业化,python够用了呗
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7281079667602883382, content: 说白点,晚出现的语言底层都是c,c++,函数,套件齐全就是慢。c++不全要自己写,难得是写代码的时间。
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7314371957528527626, content: Python就是c++基础库的精简版
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7313937205062763291, content: 简单好用,工具多,社区环境好
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7313453315387360009, content: C是打工人,Python是老板
2024-10-31 18:11:46 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7282193842115347234, content: 建议取消所有语言,只留c,再难学也比一年换一个强
2024-10-31 18:11:47 MediaCrawler INFO (core.py:185) - [DouYinCrawler.get_comments] aweme_id: 7280854932641664319 comments have all been obtained and filtered ...
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7431204786026791739, content: 求源代码
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7430467175989609268, content: 求源代码
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7413519452623192868, content: 只想学游戏脚本该怎么学python
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7316485654572008202, content: 收费吗
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7298589399850156850, content: 求源码
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7300272165453415222, content: 求源码[鼓掌][鼓掌][鼓掌]
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7319455921330897664, content: 求源码
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7426388605792174882, content: 求源代码
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7310162488493900579, content: 求源代码
2024-10-31 18:11:48 MediaCrawler INFO (init.py:106) - [store.douyin.update_dy_aweme_comment] douyin aweme comment: 7281223217397580559, content: 求源代码[流泪]
2024-10-31 18:11:49 MediaCrawler INFO (core.py:185) - [DouYinCrawler.get_comments] aweme_id: 7202432992642387233 comments have all been obtained and filtered ...
2024-10-31 18:11:49 MediaCrawler INFO (core.py:85) - [DouYinCrawler.start] Douyin Crawler finished ...
2024-10-31 18:11:49 MediaCrawler INFO (db.py:84) - [close] close mediacrawler db pool
`

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported command with python main.py --platform dy --lt qrcode --type detail, then inspect client.py, core.py, and the DouYinCrawler.get_aweme_detail and get_comments entry points. Compare their request results and logs; done means the detail request no longer reports the described account-blocked failure while comment retrieval remains functional.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.