NanmiCoder / NanmiCoder/MediaCrawler

[问题] 抖音爬取报错

Open
#679 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Python
Stars
65.3k
Forks
12.6k
PR merge metrics
No merged PRs in 30d

Description

⚠️ 提交前确认

  • 我已经仔细阅读了项目使用过程中的常见问题汇总
  • 我已经搜索并查看了已关闭的issues
  • 我确认这不是由于滑块验证码、Cookie过期、Cookie提取错误、平台风控等常见原因导致的问题

❓ 问题描述

运行

uv run main.py --platform dy --lt qrcode --type search --keywords 动画短剧

跳转二维码登录,登陆后控制台报错,报错截图在下方

🔍 使用场景

  • 目标平台: 抖音
  • 使用功能: 关键词搜索

💻 环境信息

  • 操作系统: Windows11
  • Python版本: 3.11
  • 是否使用IP代理: 是
  • 是否使用VPN翻墙软件:是
  • 目标平台(抖音/小红书/微博等): 抖音

📋 错误日志

$ uv run main.py --platform dy --lt qrcode --type search --keywords 动画短剧
2025-07-28 12:06:28 MediaCrawler INFO (core.py:72) - [DouYinCrawler] 使用标准模式启动浏览器
2025-07-28 12:06:32 MediaCrawler INFO (core.py:113) - [DouYinCrawler.search] Begin search douyin keywords
2025-07-28 12:06:32 MediaCrawler INFO (core.py:120) - [DouYinCrawler.search] Current keyword: 动画短剧
2025-07-28 12:06:32 MediaCrawler INFO (core.py:128) - [DouYinCrawler.search] Skip 0
2025-07-28 12:06:32 MediaCrawler INFO (core.py:132) - [DouYinCrawler.search] search douyin keyword: 动画短剧, page: 1
Traceback (most recent call last):
  File "E:\projectspace\MediaCrawler\main.py", line 78, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\asyncio\base_even
ts.py", line 654, in run_until_complete                                                                                           return future.result()
           ^^^^^^^^^^^^^^^
  File "E:\projectspace\MediaCrawler\main.py", line 65, in main
    await crawler.start()
  File "E:\projectspace\MediaCrawler\media_platform\douyin\core.py", line 102, in start
    await self.search()
  File "E:\projectspace\MediaCrawler\media_platform\douyin\core.py", line 135, in search
    posts_res = await self.dy_client.search_info_by_keyword(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\projectspace\MediaCrawler\media_platform\douyin\client.py", line 177, in search_info_by_keyword
    return await self.get("/aweme/v1/web/general/search/single/", query_params, headers=headers)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\projectspace\MediaCrawler\media_platform\douyin\client.py", line 112, in get
    await self.__process_req_params(uri, params, headers)
  File "E:\projectspace\MediaCrawler\media_platform\douyin\client.py", line 91, in __process_req_params
    a_bogus = await get_a_bogus(uri, query_string, post_data, headers["User-Agent"], self.playwright_page)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\projectspace\MediaCrawler\media_platform\douyin\help.py", line 51, in get_a_bogus
    return get_a_bogus_from_js(url, params, user_agent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\projectspace\MediaCrawler\media_platform\douyin\help.py", line 67, in get_a_bogus_from_js
    return douyin_sign_obj.call(sign_js_name, params, user_agent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\projectspace\MediaCrawler\.venv\Lib\site-packages\execjs\_abstract_runtime_context.py", line 37, in call
    return self._call(name, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\projectspace\MediaCrawler\.venv\Lib\site-packages\execjs\_external_runtime.py", line 92, in _call
    return self._eval("{identifier}.apply(this, {args})".format(identifier=identifier, args=args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\projectspace\MediaCrawler\.venv\Lib\site-packages\execjs\_external_runtime.py", line 78, in _eval
    return self.exec_(code)
           ^^^^^^^^^^^^^^^^
  File "E:\projectspace\MediaCrawler\.venv\Lib\site-packages\execjs\_abstract_runtime_context.py", line 18, in exec_
    return self._exec_(source)
           ^^^^^^^^^^^^^^^^^^^
  File "E:\projectspace\MediaCrawler\.venv\Lib\site-packages\execjs\_external_runtime.py", line 87, in _exec_
    output = self._exec_with_pipe(source)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\projectspace\MediaCrawler\.venv\Lib\site-packages\execjs\_external_runtime.py", line 103, in _exec_with_pipe       
    stdoutdata, stderrdata = p.communicate(input=input)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\subprocess.py", l
ine 1209, in communicate                                                                                                          stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\subprocess.py", l
ine 1622, in _communicate                                                                                                         self._stdin_write(input)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\subprocess.py", l
ine 1143, in _stdin_write                                                                                                         self.stdin.write(input)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\encodings\cp1252.
py", line 19, in encode                                                                                                           return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode characters in position 8480-8481: character maps to <undefined>

📷 错误截图

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the command in the issue, then read media_platform/douyin/help.py and media_platform/douyin/client.py around get_a_bogus_from_js and __process_req_params. Trace how execjs invokes the signing function on Windows and verify the search command completes without the reported UnicodeEncodeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.