internetarchive / internetarchive/fatcat

Spam filter

Open
#64 2 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Python
Stars
121
Forks
18
PR merge metrics
No merged PRs in 30d

Description

It would be useful to have a naive function that looks at release metadata and detects gratuitous spam. In theory upstream partner sources should be able to catch spam, but, eg, today Zenodo had more than 25,000 spam DOIs and PDFs registered:

https://fatcat.wiki/release/search?q=doi_prefix%3A10.5281+date%3A2020-11-02

Most of these have terms like `[PDF]`, EPUB, `D.O.W.N.L.O.A.D`, etc, which seem like simple statistical spam detection could find. The goal wouldn't be to make something impenetrable, just to prevent large batches from getting imported. If we had such a function in one place, we could add additional patterns over time, and reuse the function in both automated bot imports (eg, like datacite DOI metadata here) and in a review bot for human edits.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the automated bot imports, including the Datacite DOI metadata path, and the proposed review bot for human edits. Define a shared spam-detection function around the listed metadata patterns, then verify that both import and review workflows can reuse it without importing large spam batches.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.