DIYgod / DIYgod/RSSHub-Radar

Drafting a new specification of radar rules

Open
#692 17 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
7.3k
Forks
418
Avg merge
4d 13h
Merged PRs (30d)
1

Description

# Context

#635

# Motivation

Currently, the `target` field can be either 1. a **route string**; or 2. a **function**. The second use case could potentially lead to some security worry. What's more, it effectively makes radar rules unable to be converted to a JSON but remains to be a Javascript. That is, **to load remote or local-modified rules cannot avoid the usage of `eval()`**, which is strictly prohibited by some modern browsers (e.g. Firefox and Safari, total market share is [~18%](https://gs.statcounter.com/browser-market-share/desktop/worldwide)) by default. We have already seen there are two distributions ([AMO](https://addons.mozilla.org/zh-CN/firefox/addon/rsshub-radar/), [Mac App Store](https://apps.apple.com/us/app/rsshub-radar/id1610744717)) that have to disable remote rules and ignore local modifications. And no one can promise that Chrome Web Store would not ban the usage of `eval()` in the future too.

~Another reason is most radar rules use `target` **function** never need to access the page content, but just match URL query strings / do some filtering/remapping/regex replacement. These rules should have been compatible with RSSBud/RSSAid, but with the current specification, they can only use a `target` **function** which is incompatible with RSSBud/RSSAid. The new specification should change these embarrassing situations.~

Last but not least, some users indeed have their demand to use their own online rule list instead of the official one. Though the user is the only one who is responsible for their action, we should still worry about their data security since using `eval()` to load remote rules is typical **remote code execution (RCE)**. As a result, such a feature could not be added before we completely deprecated the usage of `eval()`, and the prerequisite is this issue.

# Core Idea

A typical `target` function can be described in these three workflows or there combination:

1. 1st positional parameter `param` -> post-process (string operations) -> filter -> remap
2. 2nd positional parameter `URL` -> extract qs -> ...
3. 3rd positional parameter `document` -> DOM operations -> ...

Thus, I constructed a 4-stage workflow:
* match -> post-process -> filter -> remap

But after deep diving, I think there's no need to distinguish these stages, we just need to define some actions and let rule authors chain them:
```js
matcher: {
paramName: [
{
action: ...,
...: ...,
},
{...},
{...},
...
]
}
```

## Actions

### Universal parameters
* `action`: action name
* `stopIfEmpty`: whether to stop the chain immediately if the output of the current action is an empty string? (optional, default: `true`)

### Input (must be placed at the beginning of the chain)
* URL path: `{ action: 'path' }`, `{ action: 'path', key: '...' }` (formerly `param[paramName]`)
* URL qs: `{ action: 'qs' }`, `{ action: 'qs', key: '...' }`
* raw URL: `{ action: 'URL' }`
* raw document as a string: `{ action: 'raw' }`
* `querySelector`: `{ action: 'DOM', ...(To be determined) }` (maybe we could implement some workarounds to make it chainable?)
* More? Share your ideas ☺️

### Modify
* regex matching: `{ action: 'regex', match: '...' }`, `{ action: 'regex', match: '...', matchGroup: 1 }`
* regex replacement: `{ action: 'regex', match: '...', replace: '...' }`
* ~string operation~ (regex matching/replacement should be enough)
* More? Share your ideas ☺️

### Filter
Hmm, regex replacement should be enough, but that's OK to provide a convenient action.

### Remap
```js
{
action: 'remap',
map: {
key1: 'value1',
key2: { value: 'value2', overrideTitle: 'title2' }
}
default: 'defaultValue', // optional, default: ''
}
```

# A Quick Look at the New Specification

> Need more discussion to determine more details.

Due to the support for optional parameters, #693 is a prerequisite of the new specification.

Under the new specification, the radar rule list should be exported both in `.js` and `.json`. The former, which contains all fields, is preserved for backward compatibility; the latter, in which `target` functions are filtered out and old `target` route strings are remapped to `route`, will be used by new versions of RSSHub Radar.

```js
{
'12306.cn': {
_name: '12306',
kyfw: [
{
title: '售票信息',
docs: 'https://docs.rsshub.app/travel.html#_12306',
source: ['/', '/otn/leftTicket/init'],
target: (params, url) => { // for backward compatibility, will be dropped in JSON output
const searchParams = new URL(url).searchParams;
const from = searchParams.get('fs').split(',')[0];
const to = searchParams.get('ts').split(',')[0];
const date = searchParams.get('date');
return `/12306/${date}/${from}/${to}`;
},
route: '/12306/:date/:from/:to/:type?', // `/:type?` can be omitted since here we don't match it
matcher: {
from: [
{ action: 'qs', key: 'fs' },
{ action: 'regex', match: '^([^,]+)' },
],
to: [
{ action: 'qs', key: 'ts' },
{ action: 'regex', match: '^([^,]+)' },
],
date: [
{ action: 'qs', key: 'date' }
],
},
}
],
},
}
```

A use case of remap and overrideTitle

### Old rule
```js
{
'423down.com': {
_name: '423down',
www: [
{
title: '安卓软件',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'apk') {
return '/423down/android/apk';
}
}
},
{
title: '原创软件',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'zd423') {
return '/423down/computer/originalsoft';
}
}
},
{
title: '媒体播放',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'multimedia') {
return '/423down/computer/multimedia';
}
}
},
{
title: '网页浏览',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'browser') {
return '/423down/computer/browser';
}
}
},
{
title: '图形图像',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'image') {
return '/423down/computer/image';
}
}
},
{
title: '聊天软件',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'im') {
return '/423down/computer/im';
}
}
},
{
title: '办公软件',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'work') {
return '/423down/computer/work';
}
}
},
{
title: '上传下载',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'down') {
return '/423down/computer/down';
}
}
},
{
title: '系统辅助',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'systemsoft') {
return '/423down/computer/systemsoft';
}
}
},
{
title: '系统必备',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'systemplus') {
return '/423down/computer/systemplus';
}
}
},
{
title: '安全软件',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'security') {
return '/423down/computer/security';
}
}
},
{
title: '补丁相关',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'patch') {
return '/423down/computer/patch';
}
}
},
{
title: '硬件相关',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'hardwork') {
return '/423down/computer/hardware';
}
}
},
{
title: 'windows 11',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'win11') {
return '/423down/os/win11';
}
}
},
{
title: 'windows 10',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'win10') {
return '/423down/os/win10';
}
}
},
{
title: 'windows 7',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'win7') {
return '/423down/os/win7';
}
}
},
{
title: 'windows xp',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'winxp') {
return '/423down/os/winxp';
}
}
},
{
title: 'windows pe',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'winpe') {
return '/423down/os/winpe';
}
}
}
]
},
}
```

### New rule

```js
{
'423down.com': {
_name: '423down',
www: [
{
title: '分类',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
route: '/423down/:category_and_type', // original route is `/423down/:category/:type`
matcher: {
category_and_type: [
{ action: 'path', key: 'type', stopIfEmpty: false },
{
action: 'remap',
map: {
'': { value: 'index/all', overrideTitle: '首页' },
apk: { value: 'android/apk', overrideTitle: '安卓软件' },
zd423: { value: 'computer/originalsoft', overrideTitle: '原创软件' },
multimedia: { value: 'computer/multimedia', overrideTitle: '媒体播放' },
browser: { value: 'computer/browser', overrideTitle: '网页浏览' },
image: { value: 'computer/image', overrideTitle: '图形图像' },
im: { value: 'computer/im', overrideTitle: '聊天软件' },
work: { value: 'computer/work', overrideTitle: '办公软件' },
down: { value: 'computer/down', overrideTitle: '上传下载' },
systemsoft: { value: 'computer/systemsoft', overrideTitle: '系统辅助' },
systemplus: { value: 'computer/systemplus', overrideTitle: '系统必备' },
security: { value: 'computer/security', overrideTitle: '安全软件' },
patch: { value: 'computer/patch', overrideTitle: '补丁相关' },
hardwork: { value: 'computer/hardware', overrideTitle: '硬件相关' },
win11: { value: 'os/win11', overrideTitle: 'windows 11' },
win10: { value: 'os/win10', overrideTitle: 'windows 10' },
win7: { value: 'os/win7', overrideTitle: 'windows 7' },
winxp: { value: 'os/winxp', overrideTitle: 'windows xp' },
winpe: { value: 'os/winpe', overrideTitle: 'windows pe' },
}
}
]
}
}
]
},
}
```

# More Information

Here is a filtered `radar-rules.js` containing **only** those rules using `target` **function** (based on https://github.com/DIYgod/RSSHub/blob/780031a5e3e6a43691bb877e759a152cc4c8779f/build/radar-rules.js): [radar-rules-filtered.tar.gz](https://github.com/DIYgod/RSSHub-Radar/files/8502060/radar-rules-filtered.tar.gz) (`.js` ext name is banned by GitHub, so I compressed it)

You can filter it by yourself, here's some vim magic:
```vim
:%s/\v\{\_[^{}()]+title:\_[^{}()]+},?//g
:%s/\v['"][^"']+['"]: ?\{\_[^{}]+},?//g
:%s/\v\w+: ?\[\_\s*],?//g
:%g/^\s*$/d
```


cc @NeverBehave @Cay-Zhang @LeetaoGoooo @TonyRL

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the current radar-rule format described in this issue, then read prerequisite #693 and related issue #635. Compare the existing target-function behavior with the proposed matcher actions and .js/.json exports; done requires an agreed specification that removes the need for eval() while preserving backward compatibility.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, typescript
Domain
frontend, security, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.