NanmiCoder / NanmiCoder/MediaCrawler
[Feature] 支持通过 SigCLI 自动管理登录态 Cookie
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 65.3k
- Forks
- 12.6k
- PR merge metrics
- No merged PRs in 30d
Description
背景
MediaCrawler 目前支持三种登录方式:qrcode、phone、cookie。
其中 cookie 模式需要用户手动从浏览器 DevTools 中复制 Cookie 字符串粘贴到 base_config.py 的 COOKIES 变量中。这种方式存在几个痛点:
- Cookie 过期后需要反复手动替换 — 小红书/抖音的 cookie 通常只有几小时到几天的有效期
- 无头环境下(Linux server)无法方便获取 cookie — 没有浏览器 GUI 可用
- 多平台管理繁琐 — 切换平台需要分别维护各自的 cookie
建议
集成 SigCLI 作为可选的 Cookie Provider。
SigCLI 是一个 CLI 凭证管理器,可以:
- 通过真实浏览器自动完成登录(支持扫码、SSO)
- 加密存储 Cookie,自动检测过期并刷新
- 通过
sig get <provider> --format value输出纯 cookie 字符串,可直接喂给 MediaCrawler
使用示例
以小红书为例,SigCLI 的 provider 配置:
# ~/.sig/config.yaml
providers:
xiaohongshu:
domains:
- www.xiaohongshu.com
- edith.xiaohongshu.com
entryUrl: https://www.xiaohongshu.com/explore
strategy: browser
ttl: 2h
extract:
- from: cookies
as: cookie
match: "*"
apply:
- in: header
name: Cookie
value: ${cookie}
抖音类似:
douyin:
domains:
- www.douyin.com
entryUrl: https://www.douyin.com
strategy: browser
loginMode: visible
ttl: 2h
extract:
- from: cookies
as: cookie
match: "*"
apply:
- in: header
name: Cookie
value: ${cookie}
用户只需执行一次 sig login xiaohongshu(扫码登录),之后可以:
# 获取 cookie 字符串
sig get xiaohongshu --no-redaction --format value
# 输出: a1=xxx; web_session=yyy; ...
# 或者直接注入环境变量运行 MediaCrawler
sig run xiaohongshu -- python main.py --platform xhs --lt cookie
集成方式建议
在 base_config.py 中新增一种 LOGIN_TYPE:
LOGIN_TYPE = "sigcli" # qrcode | phone | cookie | sigcli
当 LOGIN_TYPE = "sigcli" 时,自动调用 sig get <platform> --no-redaction --format value 获取 cookie,无需手动配置 COOKIES 变量。
示例实现:
import subprocess
SIGCLI_PLATFORM_MAP = {
"xhs": "xiaohongshu",
"dy": "douyin",
"ks": "kuaishou",
"bili": "bilibili",
"wb": "weibo",
}
def get_cookie_from_sigcli(platform: str) -> str:
provider = SIGCLI_PLATFORM_MAP.get(platform, platform)
result = subprocess.run(
["sig", "get", provider, "--no-redaction", "--format", "value"],
capture_output=True, text=True
)
if result.returncode == 0 and result.stdout.strip():
return result.stdout.strip()
raise RuntimeError(f"sigcli: failed to get cookie for {provider}: {result.stderr}")
优势
- 零代码改动即可使用(通过
sig run环境变量注入方式) - 支持所有 MediaCrawler 已覆盖的平台
- Cookie 过期后自动重新获取(
sig get内置 validate + refresh) - 适合 headless server 部署(
sig remote+sig sync支持将本地浏览器登录的凭证同步到远程机器) - 凭证 AES-256-GCM 加密存储,不会明文出现在配置文件中
安装:npm install -g @sigcli/cli
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in base_config.py and trace how LOGIN_TYPE and the existing qrcode, phone, and cookie flows supply cookies. Define the sigcli mode around the proposed platform mapping and subprocess call, then verify failures and empty output are reported clearly. Done means supported platforms can obtain a cookie through sig get without requiring COOKIES, while existing login modes continue to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- authentication, cli
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100