basecamp / basecamp/basecamp-sdk
Narrow check-readme-env-vars, or move it into each SDK's own test suite
- Dominant language
- Go
- Stars
- 49
- Forks
- 12
- Avg merge
- 20h 47m
- Merged PRs (30d)
- 89
Description
Follow-up from #591, which shipped this gate. Raised by the lane that built it, on the strength of a measurement rather than a preference.
## The problem
`scripts/check-readme-env-vars.py` cross-checks every env var named in a README against that SDK's source. To do that it hand-lexes **six languages** from Python — Ruby heredocs and postfix `!`, Swift bare-slash regexes, Kotlin chain whitespace, TypeScript `(process as T).env`, and so on.
Eleven review rounds landed on it. 22 findings were fixed, each with a mutation red proof and a paired negative case, growing the self-test from 105 to **155 cases**. Six remain open **by design**, each argued on-thread with evidence: Swift bare-slash regexes, Ruby heredocs, `{} / x / 2`, Kotlin chain whitespace, `(process as T).env`, HTML-commented tables.
The review is enumerating **language × syntactic form**, which has no fixed point. Round counts bear that out: 8 → 1 finding, 9 → 2, 10 → 4, 11 → 3 — and two of round eleven's were the *same* fix already made for a sibling language.
## The decisive measurement
Across all ten rounds, the `strip_noncode` mask over **977 shipping files stayed byte-identical**, and the inventory stayed at **16 variable/SDK pairs across 23 read sites**.
**No defect ever changed the gate's answer about this repo.** Every fix was precision work against hypothetical inputs. That is the signal that the design is wrong for the job, not that the implementation is unfinished.
## Why it matters
Six known false-positive sources means the gate will eventually claim a README documents a variable no source reads. That is precisely the failure that gets a gate deleted by the next person it inconveniences — and it sits on the `check:` line, so a misfire blocks CI. A gate nobody trusts is worth less than no gate.
## Two ways out
1. **Narrow it.** Drop the general-purpose lexer and check only what the evidence supports: an exact-match inventory of the 16 known pairs, failing when a README names a variable outside that set or a set member loses its read site. Small, precise, no language parsing.
2. **Move it into each SDK's own test suite**, where a real parser for that language already exists. **All six open items close for free** — Ruby's lexer knows what a heredoc is; Python's guessing at one never will.
Option 2 is the better shape and closes the open threads by construction; option 1 is cheaper.
## Not at risk either way
**The prose fixes from #591 are independent of this gate** and stand on their own — the token on-ramp, the account-ID instructions, and the three false claims (`BASECAMP_ACCOUNT_ID` that nothing read, the non-compiling Kotlin quickstart, the TS caching contradiction), plus the over-broad `XDG_CACHE_HOME` claim and the `BASECAMP_NO_KEYRING` misattribution. The gate can be narrowed, moved, or dropped without touching any of it.
Contributor guide
Research direction
Start with scripts/check-readme-env-vars.py and its 155-case self-test, then review the 16 variable/SDK pairs across 23 read sites. Choose between narrowing the check to the measured inventory or moving validation into each SDK's test suite; done means the CI gate avoids the unsupported general-purpose lexing while preserving the intended README checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kotlin, python, ruby, swift, typescript
- Domain
- ci-cd, tooling
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100