BOHICA-LABS / BOHICA-LABS/writescore

Strip YAML frontmatter before analysis

Open
#48 1 comment 0 reactions 0 assignees View on GitHub
stale
Dominant language
Python
Stars
2
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Problem

When analyzing markdown files with YAML frontmatter (common in static site generators like Jekyll, Hugo, etc.), the tool includes the frontmatter in its semantic coherence analysis. This causes false positives:

- Frontmatter (lines between `---` delimiters) is detected as "disconnected paragraphs"
- The Material Purpose or similar metadata sections are flagged as having "weak transitions" to the article content
- This artificially lowers semantic coherence scores

## Example

A file structured like:
```markdown
---
title: "Article Title"
status: draft
---

## Material Purpose (for Blue Quill Submission)

This article addresses...

---

# Article Title

Your organization has more...
```

Gets flagged with:
- `Para 1->2 (too disconnected, sim=0.24)` - frontmatter to heading
- `Para 2->3 (too disconnected, sim=0.08)` - heading to Material Purpose
- `Para 4->5 (sim=0.01)` - Material Purpose to second `---` delimiter

## Proposed Solution

Before analysis, strip:
1. YAML frontmatter (content between opening `---` and closing `---` at the start of the file)
2. Optionally: common metadata sections like "Material Purpose" that are clearly not article content

This would give more accurate scores for the actual article content that will be published.

## Impact

Currently seeing ~3.8 pts potential improvement blocked by frontmatter issues in semantic coherence scoring.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.