NanmiCoder / NanmiCoder/MediaCrawler

[Feature] 支持通过 SigCLI 自动管理登录态 Cookie

Open
#896 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
65.3k
Forks
12.6k
PR merge metrics
No merged PRs in 30d

Description

背景

MediaCrawler 目前支持三种登录方式:qrcodephonecookie

其中 cookie 模式需要用户手动从浏览器 DevTools 中复制 Cookie 字符串粘贴到 base_config.pyCOOKIES 变量中。这种方式存在几个痛点:

  1. Cookie 过期后需要反复手动替换 — 小红书/抖音的 cookie 通常只有几小时到几天的有效期
  2. 无头环境下(Linux server)无法方便获取 cookie — 没有浏览器 GUI 可用
  3. 多平台管理繁琐 — 切换平台需要分别维护各自的 cookie

建议

集成 SigCLI 作为可选的 Cookie Provider。

SigCLI 是一个 CLI 凭证管理器,可以:

  • 通过真实浏览器自动完成登录(支持扫码、SSO)
  • 加密存储 Cookie,自动检测过期并刷新
  • 通过 sig get <provider> --format value 输出纯 cookie 字符串,可直接喂给 MediaCrawler

使用示例

以小红书为例,SigCLI 的 provider 配置:

# ~/.sig/config.yaml
providers:
  xiaohongshu:
    domains:
      - www.xiaohongshu.com
      - edith.xiaohongshu.com
    entryUrl: https://www.xiaohongshu.com/explore
    strategy: browser
    ttl: 2h
    extract:
      - from: cookies
        as: cookie
        match: "*"
    apply:
      - in: header
        name: Cookie
        value: ${cookie}

抖音类似:

  douyin:
    domains:
      - www.douyin.com
    entryUrl: https://www.douyin.com
    strategy: browser
    loginMode: visible
    ttl: 2h
    extract:
      - from: cookies
        as: cookie
        match: "*"
    apply:
      - in: header
        name: Cookie
        value: ${cookie}

用户只需执行一次 sig login xiaohongshu(扫码登录),之后可以:

# 获取 cookie 字符串
sig get xiaohongshu --no-redaction --format value
# 输出: a1=xxx; web_session=yyy; ...

# 或者直接注入环境变量运行 MediaCrawler
sig run xiaohongshu -- python main.py --platform xhs --lt cookie

集成方式建议

base_config.py 中新增一种 LOGIN_TYPE

LOGIN_TYPE = "sigcli"  # qrcode | phone | cookie | sigcli

LOGIN_TYPE = "sigcli" 时,自动调用 sig get <platform> --no-redaction --format value 获取 cookie,无需手动配置 COOKIES 变量。

示例实现:

import subprocess

SIGCLI_PLATFORM_MAP = {
    "xhs": "xiaohongshu",
    "dy": "douyin",
    "ks": "kuaishou",
    "bili": "bilibili",
    "wb": "weibo",
}

def get_cookie_from_sigcli(platform: str) -> str:
    provider = SIGCLI_PLATFORM_MAP.get(platform, platform)
    result = subprocess.run(
        ["sig", "get", provider, "--no-redaction", "--format", "value"],
        capture_output=True, text=True
    )
    if result.returncode == 0 and result.stdout.strip():
        return result.stdout.strip()
    raise RuntimeError(f"sigcli: failed to get cookie for {provider}: {result.stderr}")

优势

  • 零代码改动即可使用(通过 sig run 环境变量注入方式)
  • 支持所有 MediaCrawler 已覆盖的平台
  • Cookie 过期后自动重新获取(sig get 内置 validate + refresh)
  • 适合 headless server 部署(sig remote + sig sync 支持将本地浏览器登录的凭证同步到远程机器)
  • 凭证 AES-256-GCM 加密存储,不会明文出现在配置文件中

安装:npm install -g @sigcli/cli

文档:https://github.com/sigcli/sigcli

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in base_config.py and trace how LOGIN_TYPE and the existing qrcode, phone, and cookie flows supply cookies. Define the sigcli mode around the proposed platform mapping and subprocess call, then verify failures and empty output are reported clearly. Done means supported platforms can obtain a cookie through sig get without requiring COOKIES, while existing login modes continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
authentication, cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.