agenticsorg / agenticsorg/community-projects
fix(accuracy): OIA auto-classifier produces near-universal false positives from .github boilerplate
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 1
- Avg merge
- 1m
- Merged PRs (30d)
- 1
Description
The heuristic classifier (scripts/classify-oia.mjs) builds a single corpus from the repo file tree + README + topics + language, then regex-matches signal groups. Because raw file paths are in the corpus, GitHub boilerplate that exists on most repos drives two of the most "reliable" signals.
Reproduced on sindresorhus/slugify (a trivial string utility):
- classifies as L7 Orchestration & Workflow (presence) because
.github/workflows/and.github/workflows/main.ymlmatch/workflow/ - gets a security span because
.github/security.mdmatches/security/
Secondary: several groups lack word boundaries, so common English inflates hits (e.g. /knowledge/ is a substring of "acknowledged").
Impact: L7 and the security span are close to noise across the matrix; self-service submitters can also salt a README to place themselves on flattering layers.
Proposed fix (see companion patch):
- Exclude boilerplate from the corpus before matching:
.github/,node_modules, lockfiles,LICENSE,SECURITY.md,CODE_OF_CONDUCT. - Add
\bboundaries to substring-prone tokens (knowledge,train,serve,vector). - Weight README + topics above raw tree paths rather than a flat equal-footing concatenation.
Must be applied to BOTH the server classifier and the retest() twin in docs/oia-matrix.html (see the twin-sync issue).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in scripts/classify-oia.mjs by tracing how the corpus is assembled and how signal groups are matched, then compare the retest() twin in docs/oia-matrix.html. Verify the boilerplate exclusions, word boundaries, and README/topics weighting against the slugify reproduction, and ensure both classifiers stay synchronized.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100