basecamp / basecamp/basecamp-sdk

Narrow check-readme-env-vars, or move it into each SDK's own test suite

Open
#611 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
49
Forks
12
Avg merge
20h 47m
Merged PRs (30d)
89

Description

Follow-up from #591, which shipped this gate. Raised by the lane that built it, on the strength of a measurement rather than a preference.

## The problem

`scripts/check-readme-env-vars.py` cross-checks every env var named in a README against that SDK's source. To do that it hand-lexes **six languages** from Python — Ruby heredocs and postfix `!`, Swift bare-slash regexes, Kotlin chain whitespace, TypeScript `(process as T).env`, and so on.

Eleven review rounds landed on it. 22 findings were fixed, each with a mutation red proof and a paired negative case, growing the self-test from 105 to **155 cases**. Six remain open **by design**, each argued on-thread with evidence: Swift bare-slash regexes, Ruby heredocs, `{} / x / 2`, Kotlin chain whitespace, `(process as T).env`, HTML-commented tables.

The review is enumerating **language × syntactic form**, which has no fixed point. Round counts bear that out: 8 → 1 finding, 9 → 2, 10 → 4, 11 → 3 — and two of round eleven's were the *same* fix already made for a sibling language.

## The decisive measurement

Across all ten rounds, the `strip_noncode` mask over **977 shipping files stayed byte-identical**, and the inventory stayed at **16 variable/SDK pairs across 23 read sites**.

**No defect ever changed the gate's answer about this repo.** Every fix was precision work against hypothetical inputs. That is the signal that the design is wrong for the job, not that the implementation is unfinished.

## Why it matters

Six known false-positive sources means the gate will eventually claim a README documents a variable no source reads. That is precisely the failure that gets a gate deleted by the next person it inconveniences — and it sits on the `check:` line, so a misfire blocks CI. A gate nobody trusts is worth less than no gate.

## Two ways out

1. **Narrow it.** Drop the general-purpose lexer and check only what the evidence supports: an exact-match inventory of the 16 known pairs, failing when a README names a variable outside that set or a set member loses its read site. Small, precise, no language parsing.
2. **Move it into each SDK's own test suite**, where a real parser for that language already exists. **All six open items close for free** — Ruby's lexer knows what a heredoc is; Python's guessing at one never will.

Option 2 is the better shape and closes the open threads by construction; option 1 is cheaper.

## Not at risk either way

**The prose fixes from #591 are independent of this gate** and stand on their own — the token on-ramp, the account-ID instructions, and the three false claims (`BASECAMP_ACCOUNT_ID` that nothing read, the non-compiling Kotlin quickstart, the TS caching contradiction), plus the over-broad `XDG_CACHE_HOME` claim and the `BASECAMP_NO_KEYRING` misattribution. The gate can be narrowed, moved, or dropped without touching any of it.

Contributor guide

Open the contributing guide

Research direction

Start with scripts/check-readme-env-vars.py and its 155-case self-test, then review the 16 variable/SDK pairs across 23 read sites. Choose between narrowing the check to the measured inventory or moving validation into each SDK's test suite; done means the CI gate avoids the unsupported general-purpose lexing while preserving the intended README checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
kotlin, python, ruby, swift, typescript
Domain
ci-cd, tooling
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.