NanmiCoder / NanmiCoder/MediaCrawler
[问题] 抖音爬取报错
Open
Nobody has claimed this yet.
question
- Dominant language
- Python
- Stars
- 65.3k
- Forks
- 12.6k
- PR merge metrics
- No merged PRs in 30d
Description
⚠️ 提交前确认
- 我已经仔细阅读了项目使用过程中的常见问题汇总
- 我已经搜索并查看了已关闭的issues
- 我确认这不是由于滑块验证码、Cookie过期、Cookie提取错误、平台风控等常见原因导致的问题
❓ 问题描述
运行
uv run main.py --platform dy --lt qrcode --type search --keywords 动画短剧
跳转二维码登录,登陆后控制台报错,报错截图在下方
🔍 使用场景
- 目标平台: 抖音
- 使用功能: 关键词搜索
💻 环境信息
- 操作系统: Windows11
- Python版本: 3.11
- 是否使用IP代理: 是
- 是否使用VPN翻墙软件:是
- 目标平台(抖音/小红书/微博等): 抖音
📋 错误日志
$ uv run main.py --platform dy --lt qrcode --type search --keywords 动画短剧
2025-07-28 12:06:28 MediaCrawler INFO (core.py:72) - [DouYinCrawler] 使用标准模式启动浏览器
2025-07-28 12:06:32 MediaCrawler INFO (core.py:113) - [DouYinCrawler.search] Begin search douyin keywords
2025-07-28 12:06:32 MediaCrawler INFO (core.py:120) - [DouYinCrawler.search] Current keyword: 动画短剧
2025-07-28 12:06:32 MediaCrawler INFO (core.py:128) - [DouYinCrawler.search] Skip 0
2025-07-28 12:06:32 MediaCrawler INFO (core.py:132) - [DouYinCrawler.search] search douyin keyword: 动画短剧, page: 1
Traceback (most recent call last):
File "E:\projectspace\MediaCrawler\main.py", line 78, in <module>
asyncio.get_event_loop().run_until_complete(main())
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\asyncio\base_even
ts.py", line 654, in run_until_complete return future.result()
^^^^^^^^^^^^^^^
File "E:\projectspace\MediaCrawler\main.py", line 65, in main
await crawler.start()
File "E:\projectspace\MediaCrawler\media_platform\douyin\core.py", line 102, in start
await self.search()
File "E:\projectspace\MediaCrawler\media_platform\douyin\core.py", line 135, in search
posts_res = await self.dy_client.search_info_by_keyword(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\projectspace\MediaCrawler\media_platform\douyin\client.py", line 177, in search_info_by_keyword
return await self.get("/aweme/v1/web/general/search/single/", query_params, headers=headers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\projectspace\MediaCrawler\media_platform\douyin\client.py", line 112, in get
await self.__process_req_params(uri, params, headers)
File "E:\projectspace\MediaCrawler\media_platform\douyin\client.py", line 91, in __process_req_params
a_bogus = await get_a_bogus(uri, query_string, post_data, headers["User-Agent"], self.playwright_page)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\projectspace\MediaCrawler\media_platform\douyin\help.py", line 51, in get_a_bogus
return get_a_bogus_from_js(url, params, user_agent)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\projectspace\MediaCrawler\media_platform\douyin\help.py", line 67, in get_a_bogus_from_js
return douyin_sign_obj.call(sign_js_name, params, user_agent)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\projectspace\MediaCrawler\.venv\Lib\site-packages\execjs\_abstract_runtime_context.py", line 37, in call
return self._call(name, *args)
^^^^^^^^^^^^^^^^^^^^^^^
File "E:\projectspace\MediaCrawler\.venv\Lib\site-packages\execjs\_external_runtime.py", line 92, in _call
return self._eval("{identifier}.apply(this, {args})".format(identifier=identifier, args=args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\projectspace\MediaCrawler\.venv\Lib\site-packages\execjs\_external_runtime.py", line 78, in _eval
return self.exec_(code)
^^^^^^^^^^^^^^^^
File "E:\projectspace\MediaCrawler\.venv\Lib\site-packages\execjs\_abstract_runtime_context.py", line 18, in exec_
return self._exec_(source)
^^^^^^^^^^^^^^^^^^^
File "E:\projectspace\MediaCrawler\.venv\Lib\site-packages\execjs\_external_runtime.py", line 87, in _exec_
output = self._exec_with_pipe(source)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\projectspace\MediaCrawler\.venv\Lib\site-packages\execjs\_external_runtime.py", line 103, in _exec_with_pipe
stdoutdata, stderrdata = p.communicate(input=input)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\subprocess.py", l
ine 1209, in communicate stdout, stderr = self._communicate(input, endtime, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\subprocess.py", l
ine 1622, in _communicate self._stdin_write(input)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\subprocess.py", l
ine 1143, in _stdin_write self.stdin.write(input)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\encodings\cp1252.
py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode characters in position 8480-8481: character maps to <undefined>
📷 错误截图
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the command in the issue, then read media_platform/douyin/help.py and media_platform/douyin/client.py around get_a_bogus_from_js and __process_req_params. Trace how execjs invokes the signing function on Windows and verify the search command completes without the reported UnicodeEncodeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100