internetarchive / internetarchive/fatcat
Spam filter
- Dominant language
- Python
- Stars
- 121
- Forks
- 18
- PR merge metrics
- No merged PRs in 30d
Description
It would be useful to have a naive function that looks at release metadata and detects gratuitous spam. In theory upstream partner sources should be able to catch spam, but, eg, today Zenodo had more than 25,000 spam DOIs and PDFs registered:
https://fatcat.wiki/release/search?q=doi_prefix%3A10.5281+date%3A2020-11-02
Most of these have terms like `[PDF]`, EPUB, `D.O.W.N.L.O.A.D`, etc, which seem like simple statistical spam detection could find. The goal wouldn't be to make something impenetrable, just to prevent large batches from getting imported. If we had such a function in one place, we could add additional patterns over time, and reuse the function in both automated bot imports (eg, like datacite DOI metadata here) and in a review bot for human edits.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the automated bot imports, including the Datacite DOI metadata path, and the proposed review bot for human edits. Define a shared spam-detection function around the listed metadata patterns, then verify that both import and review workflows can reuse it without importing large spam batches.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100