【RSSHub Gateway】一个简单的解决方案，适用于Image Referer和严格反爬路由

### 这是一个什么样的功能？

一个简易的RSSHub Gateway，提供两个核心功能

1. 不可用路由自动切换实例
2. 图片代理

我觉得这个功能可以融合进RSSHub中，提升RSSHub的可用性

### 这个功能可以解决什么问题？

https://github.com/DIYgod/RSSHub/issues/11096
https://github.com/DIYgod/RSSHub/issues/14903
https://github.com/DIYgod/RSSHub/issues/12599

越来越多的网站对于图片都添加了防盗链，虽然可以通过浏览器插件，强制修改referer。 但是在移动端、客户端场景，该方案很难实施。

对于一些反扒严格的路由，如sehuatang，对于单个自建实例容易出现cloudflare验证，结合公共实例，希望能通过自动切换后端实例提升路由的可用性。

### 额外描述

我贴出我的一个简易python实现

运行原理

该Gateway提供了两个路由
1. `/rsshub/(*any)` ，对于 `(*any)`路由，遍历`websiteInstances`，直到第一个200响应
2. `/image`，该路由接收两个query参数`url`和`referer`，分别是图片地址和Referer。该路由无需手动配置，见下面使用方法3。

如何使用？假设当前Gateway运行的URL是http://192.168.1.2:8080
1. 对该Gateway添加网络代理，配置环境变量`PROXY_URI=http://ip:port`
2. 路由高可用，原先`/weibo/user/:uid/:routeParams?` -> `http://192.168.1.2:8080/rsshub/weibo/user/:uid/:routeParams?`
3. 路由高可用+图片代理：`http://192.168.1.2:8080/rsshub/sehuatang/103?_image_proxy=1&_image_proxy_referer=https://www.sehuatang.net` 两个关键 query参数`_image_proxy`针对当前路由开启图片代理，`_image_proxy_referer` 表示访问该图片需要的referer。当`_image_proxy=1`时，Gateway程序会对响应结果中的标签通过正则方式替换成上面的`http://192.168.1.2:8080/image?url=原先的图片地址&referer=${_image_proxy_referer}`。

pip依赖安装
```
requests
fastapi
uvicorn
cachetools
httpx
```

```python
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse,Response
import requests
import os
import urllib.parse
from cachetools import LRUCache, TTLCache
from typing import Any
import httpx
import logging
import sys
import re

# 创建一个LRU缓存实例，例如：缓存100个最近使用的条目，并且每个条目在5分钟内有效
cache = TTLCache(maxsize=100, ttl=5 * 60)
app = FastAPI()

# 设置日志格式，包含文件名、行号和方法名
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(filename)s:%(lineno)d - %(funcName)s - %(message)s')
# 创建一个stream handler并设置格式
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(formatter)

# 获取logger并设置日志级别和handler
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
logger.addHandler(handler)

@app.get('/image')
async def proxy_request(request: Request):
query_string = request.url.query
# 解码原始查询字符串以正确处理转义的 &
query_params = urllib.parse.parse_qs(query_string)

url_param = query_params.get('url', [''])[0]
# 尝试从缓存中获取结果
cached_response = cache.get(url_param)
if cached_response is not None:
headers, content = cached_response
return StreamingResponse(iter([content]), status_code=200, headers=headers)

referer_env = os.environ.get('DEFAULT_REFERER', '')
referer_param = query_params.get('referer', [referer_env])[0]

logger.info(f"Proxying request to URL: {url_param}")
logger.info(f"Referer: {referer_param}")

user_agent_env = os.environ.get('USER_AGENT_HEADER')
if user_agent_env is None:
user_agent_header = request.headers.get('user-agent', '')
else:
user_agent_header = user_agent_env

proxy_uri = os.environ.get('PROXY_URI', None)

async with httpx.AsyncClient(proxy=proxy_uri) as client:
response = await client.get(url_param, headers={'referer': referer_param, 'user-agent': user_agent_header})
# 确保完整读取response.content
response_content = response.content
content_length = len(response_content)

headers = response.headers.copy()
headers['Content-Length'] = str(content_length) # 更新Content-Length为实际长度
# 只缓存HTTP状态码为200的响应
if response.status_code == 200:
cache[url_param] = (headers, response_content)

return StreamingResponse(
iter([response_content]),
status_code=response.status_code,
headers=headers,
)

# 假设你有一系列网站实例
websiteInstances = [
"http://172.17.0.1:1200",
"https://rsshub.app",
"https://rsshub.rssforever.com",
"https://rsshub.feeded.xyz",
"https://hub.slarker.me",
"https://rsshub.liumingye.cn",
"https://rsshub-instance.zeabur.app",
"https://rsshub.pseudoyu.com",
"https://rsshub.friesport.ac.cn",
"https://rsshub.atgw.io",
"https://rsshub.rss.tips",
"https://rsshub.mubibai.com",
"https://rsshub.ktachibana.party",
"https://rsshub.woodland.cafe",
"https://rsshub.aierliz.xyz"
]

IMAGE_PROXY_KEY = "_image_proxy"
IMAGE_PROXY_REFERER_KEY = "_image_proxy_referer"
async def forward_request(request: Request, path: str):
proxy_uri = os.environ.get('PROXY_URI', None)
if not path.startswith('/'):
path = f"/{path}"
query_params = request.query_params
async with httpx.AsyncClient(proxy=proxy_uri) as client:
for website in websiteInstances:
full_url = f"{website}{path}"
if query_params:
full_url += '?' + '&'.join([f"{k}={v}" for k, v in query_params.items()])
try:
response = await client.get(full_url)
if response.status_code == 200:
content = str(response.content.decode('utf-8'))
if IMAGE_PROXY_KEY in query_params.keys():
content = replace_img_with_template(content, f"{request.url.hostname}:{request.url.port}", query_params.get(IMAGE_PROXY_REFERER_KEY, ''))
return Response(
content=content,
media_type=response.headers["content-type"],
)
except Exception as e:
logger.error(f"Error occurred while requesting {full_url}, err: {str(e)}")
else:
if response.status_code != 200:
logger.info(f"Non-200 status code ({response.status_code}) from {full_url}")

return None

@app.get("/rsshub/{path:path}")
async def rsshub_handler(request: Request, path: str):
result = await forward_request(request, path)
if result is not None:
return result
else:
return {"error": "No website instance returned a 200 status code"}

def replace_img_with_template(content: str, host: str, referer) -> bytes:
pattern = r'