matomo-org / matomo-org/device-detector

Detect bots from HTTP_FROM

Open
#7,929 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PHP
Stars
3.5k
Forks
513
Avg merge
1d 21h
Merged PRs (30d)
11

Description

I think this should be implemented, and I have three reasons for this:

  1. User agents change and it's hard to keep up will all bots, worldwide, especially if they don't get reported so we can add them, and also we can detect them right away.
  2. Bing uses Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 Edg/130.0.0.0 and HTTP_FROM is bingbot(at)microsoft.com.
  3. Bots are required to send HTTP_FROM https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/From

If HTTP_FROM is detected then it should be treated as generic bot, then further check for specific bot name.

  • "<?=print(9347655345-4954366);?>"
    Mozilliqa"<?=print(9347655345-4954366);?>"

  • TGVnaXRpbWF0ZSBsaW5rIHRyYWNrZXI=
    Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0

  • bingbot(at)microsoft.com
    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 Edg/130.0.0.0

  • crawler@alexa.com
    ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)

  • crawler@rois.ac.jp
    Mozilla/5.0 (compatible; Cotoyogi/4.0; +https://ds.rois.ac.jp/center8/crawler/

  • googlebot(at)googlebot.com
    Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

  • gptbot(at)openai.com
    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)

  • morten@cs.umn.edu
    SuggestBot/1.0

  • oai-searchbot@openai.com
    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot

  • pigafetta-bot(at)visual-seo.com
    Mozilla 5.0 (compatible; Pigafetta/0.5; +http://visual-seo.com/Pigafetta-Bot)

  • robot@seokicks.de
    Mozilla/5.0 (compatible; SEOkicks; +https://www.seokicks.de/robot.html)

  • root@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.oast.site
    Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 root@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.oast.site

  • support@search.yandex.ru
    Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

  • tech@babbar.tech
    Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)

  • the.knowledge.ai@gmail.com
    The Knowledge AI

  • wc@verisign.com
    Mozilla/5.0 (compatible; ips-agent)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or code entry points are named. Start by tracing the existing bot-detection path and how request headers and user-agent strings are handled, then determine where HTTP_FROM should trigger generic bot detection and where the listed addresses should identify specific bots. Done means the proposed cases are covered without relying only on changing user-agent strings.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
backend, security
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.