thunderbird / thunderbird/github-action-thunderbird-aaq
Detect and log JavaScript/bot challenges in the SUMO fetch scripts
Nobody has claimed this yet.
- Dominant language
- Ruby
- Stars
- 0
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
Problem
The two fetch scripts pull JSON from support.mozilla.org/api/2/... through the shared getKitsuneResponse helper. If SUMO ever sits behind a JavaScript/bot challenge (Cloudflare "Just a moment…", /cdn-cgi/challenge-platform/, "enable JavaScript and cookies to continue", etc.), the endpoint returns an HTML page instead of JSON.
Today that path is silent and misleading:
JSON.parse(result.body)raisesJSON::ParserError- it is retried 3× (pointless — re-requesting the same URL re-serves the same challenge), then
getKitsuneResponsereturnsnil - callers treat
nilas "no data for this window" → they write an empty CSV (or skip a question's answers) and the run looks like a normal quiet day
So a challenge is indistinguishable from a genuinely empty result.
Goal
Detect challenge responses, log them to a dedicated auditable file (plus STDERR), and abort the day's run with a non-zero exit so the failure is visible rather than masquerading as empty data.
Implementation (done in da6ea03074)
get-kitsune-response.rb (only place with access to result.body + headers):
- Added a
JavascriptChallengeErrorclass carrying url, response_code, body snippet. - Added a
javascript_challenge?(body, content_type)detector matching well-known markers (just a moment,cdn-cgi/challenge-platform,cf-browser-verification,_cf_chl_opt,checking your browser,enable javascript and cookies, etc.), with HTML content-type +<noscript>/challengeas a corroborating signal. - Added a shared
log_javascript_challenge(logger, error, context)writer that appends one line tojavascript-challenges.log(resolved via__dir__so it always lands at repo root regardless of cwd) and mirrors tologger.error. - In
rescue JSON::ParserError, checks for a challenge first and raisesJavascriptChallengeErrorinstead of pointlessly retrying-then-returning-nil.
get-tb-creator-answers-questions-for-arbitrary-time-period.rb and get-tb-answers-from-questions-file-for-arbitrary-time-period.rb:
- Wrapped the fetch loop in
rescue JavascriptChallengeError => e→log_javascript_challenge(...)thenexit 1. - Genuine per-question
nilmisses keep their current "log + continue" behavior; only an actual challenge aborts.
Follow-ups
- No new dependencies (stdlib
File/Time+ existing Typhoeusresult.headers). - Decide whether
javascript-challenges.logshould be committed for audit (and if so confirm the GitHub Actions workflows stage it) or.gitignored (Actions-log only).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review get-kitsune-response.rb and the two named fetch scripts first; inspect commit da6ea03074 and the GitHub Actions workflows to understand where javascript-challenges.log is expected to go. Done means resolving whether the log is committed or ignored and confirming the workflows preserve the chosen audit behavior; the challenge handling itself is described as implemented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, ruby
- Domain
- api, devops
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100