BOHICA-LABS / BOHICA-LABS/writescore
Exclude markdown headings from paragraph coherence analysis
- Dominant language
- Python
- Stars
- 2
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
The semantic coherence analysis treats markdown headings (`#`, `##`, `###`, etc.) as paragraphs and flags transitions between headings and content as "too disconnected."
Example from analysis:
```
Para 1->2 (sim=0.05): '# Building an Attack Surface Discovery Workflow...' -> 'Your organization owns more...'
Para 5->6 (sim=0.29): 'Attackers don't share your blind spots...' -> '## The Problem: You Can't Secure...'
```
Headings are structural labels, not content paragraphs. They will **always** have low semantic similarity to surrounding text because:
- They're short summaries, not full ideas
- They mark section boundaries, not idea continuity
- Their purpose is navigation, not discourse flow
## Proposed Solution
Before computing paragraph transitions:
1. Identify lines starting with `#` as headings (not paragraphs)
2. Exclude heading → paragraph and paragraph → heading transitions from coherence scoring
3. Only analyze transitions between actual prose paragraphs
Alternatively, treat headings as section delimiters that "reset" the coherence expectation - the first paragraph after a heading shouldn't need to connect to the last paragraph of the previous section.
## Impact
Articles with proper heading structure get penalized for having headings. This artificially lowers semantic coherence scores and triggers "weak transition" warnings for structurally correct documents.
Contributor guide
Assessment
This issue has not been assessed yet.