agenticsorg / agenticsorg/community-projects

fix(accuracy): OIA auto-classifier produces near-universal false positives from .github boilerplate

Open
#48 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

priority/p1 type/bug
Dominant language
JavaScript
Stars
3
Forks
1
Avg merge
1m
Merged PRs (30d)
1

Description

The heuristic classifier (scripts/classify-oia.mjs) builds a single corpus from the repo file tree + README + topics + language, then regex-matches signal groups. Because raw file paths are in the corpus, GitHub boilerplate that exists on most repos drives two of the most "reliable" signals.

Reproduced on sindresorhus/slugify (a trivial string utility):

  • classifies as L7 Orchestration & Workflow (presence) because .github/workflows/ and .github/workflows/main.yml match /workflow/
  • gets a security span because .github/security.md matches /security/

Secondary: several groups lack word boundaries, so common English inflates hits (e.g. /knowledge/ is a substring of "acknowledged").

Impact: L7 and the security span are close to noise across the matrix; self-service submitters can also salt a README to place themselves on flattering layers.

Proposed fix (see companion patch):

  1. Exclude boilerplate from the corpus before matching: .github/, node_modules, lockfiles, LICENSE, SECURITY.md, CODE_OF_CONDUCT.
  2. Add \b boundaries to substring-prone tokens (knowledge, train, serve, vector).
  3. Weight README + topics above raw tree paths rather than a flat equal-footing concatenation.

Must be applied to BOTH the server classifier and the retest() twin in docs/oia-matrix.html (see the twin-sync issue).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in scripts/classify-oia.mjs by tracing how the corpus is assembled and how signal groups are matched, then compare the retest() twin in docs/oia-matrix.html. Verify the boilerplate exclusions, word boundaries, and README/topics weighting against the slugify reproduction, and ensure both classifiers stay synchronized.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.