adobe / adobe/skills

find-test-content: no rate-limit handling causes silent false negatives under HTTP 429

Open
#351 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
JavaScript
Stars
182
Forks
73
Avg merge
1d 5h
Merged PRs (30d)
30

Description

## Summary

The `find-test-content` skill's crawler (`scripts/find-block-content.js`) has no rate-limit handling. When the target site returns HTTP 429 (Too Many Requests), throttled pages are silently counted as "block not found", producing incorrect results with no warning to the user.

## Problem 1: No rate-limit handling

`findBlockInPages()` (line 142) maintains 10 concurrent requests via a `Set` + `Promise.race` pool. Nothing in the file inspects HTTP 429, reads `Retry-After`, backs off, or throttles. Against a fast CDN edge, 10-way concurrency sustains a request rate high enough to trigger server-side rate limiting, and the crawler continues at full speed.

Searching the file for `429`, `Retry-After`, `backoff`, `sleep`, `delay`, `throttle`, or `retry` returns zero matches.

## Problem 2: Rate-limited pages silently miscounted as "block not found" (correctness bug)

In `pageContainsBlock()` (approx. line 88):

```js
const res = await fetch(url, { headers: { 'User-Agent': USER_AGENT } });
if (!res.ok) {
return null;
}
```

`null` is the same value returned when a page genuinely does not contain the block (approx. line 109). The caller in `findBlockInPages` treats `null` as "no match" and adds nothing to `matches`. No warning is emitted.

This means a 429 (or 503, or any transient error) is **indistinguishable from a real negative**. The skill then prints a confident, complete-looking result that is actually missing every page it was throttled on. These are **false negatives presented as fact**.

Additionally, the `catch` block (approx. line 118) swallows all exceptions and returns `null`, so even network errors are silently treated as "block absent".

## Problem 3: fetchQueryIndex truncation on error

`fetchQueryIndex()` (line 39) paginates the query index. Its `catch` block:

```js
catch (err) {
console.error(`Error fetching query index: ${err.message}`);
more = false;
}
```

A single 429 mid-pagination sets `more = false`, silently truncating the path list. The caller receives a partial inventory and proceeds as though it were complete. The only signal is a stderr line that an LLM caller may not surface.

## Production measurements

Over the 7-day window ending 2026-09-14, CDN logs for the `AdobeSkills/1.0 (skill:find-test-content)` user agent show:

- **74,105 total requests across 11 hosts** (first seen 2026-09-07 13:38:38 UTC, last seen 2026-09-11 22:53:28 UTC).
- The host `develop--nrf-eds-dev--us-norton-rose-fulbright.aem.page` received **66,800 requests**, of which **34,600 returned HTTP 429** and 32,200 returned 200 -- a **51.8% rejection rate**.
- Peak observed rate: **9,100 requests in a single minute (~150 req/s)** at 2026-09-11 22:52 UTC.
- Daily breakdown for that host: 2026-09-07 6,100 (0 rejected); 09-08 15,101 (0); 09-09 600 (0); 09-10 4 (0); **09-11 52,300 requests of which 34,600 were 429s**.
- This is a **customer development site** (`.aem.page` preview tier), not an Adobe-owned environment.

On 2026-09-11, more than half the requests were rejected, and every rejected page was silently treated as "does not contain the block". Any result the operator received was wrong, and nothing in the output indicated this.

## Why this matters

Problem 1 is a politeness/citizenship issue. Problem 2 is a **correctness bug**: the skill produces silently wrong answers. Users trust the output ("Found 12 pages") without knowing that dozens of additional pages were rate-limited and excluded.

## Additional notes

- The default `concurrency = 10` is not overridable from the CLI entry point (`main()` reads only `argv[2]` and `argv[3]`).
- `SKILL.md` documents no rate-limit behavior, concurrency guidance, or incomplete-result warnings.

## Proposed remedy

1. Detect HTTP 429/503 and retry with exponential backoff, honoring `Retry-After` when present.
2. Distinguish three outcomes in `pageContainsBlock`: found, genuinely absent, and could-not-fetch.
3. Report incomplete results explicitly: if any pages could not be checked after retries, state the count so the user knows the result is partial.
4. Apply the same retry logic to `fetchQueryIndex` pagination.
5. Lower default concurrency and/or add inter-request delay; make it configurable via CLI.
6. Update `SKILL.md` to document the new behavior.

Contributor guide

Open the contributing guide

Research direction

Start with scripts/find-block-content.js, reading pageContainsBlock(), findBlockInPages(), fetchQueryIndex(), and the main() argument handling. Trace how HTTP failures and null results flow into matches, then verify that retries, incomplete-result reporting, pagination errors, and configurable request pacing behave as described; update SKILL.md for the documented behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.