BOHICA-LABS / BOHICA-LABS/writescore

bug: Dimension weight test expectations don't match actual scoring behavior

Open
#9 1 comment 0 reactions 0 assignees View on GitHub
area: scoring good first issue priority: medium scope: tech-debt type: bug
Dominant language
Python
Stars
2
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Summary

Four unit tests in `test_dual_score_calculator.py` fail because their hardcoded expected values don't match the actual dimension weights used by the scoring system. The tests assert specific score values based on assumed weights (18.1 for predictability, 12.8 for advanced_lexical), but the actual calculated scores differ by ~2.5-3%.

## Severity

**Medium** - Tests fail but core functionality works correctly. The scoring system produces valid scores; only the test expectations are incorrect.

## Failing Tests

**File:** `tests/unit/scoring/test_dual_score_calculator.py`

| Test | Expected | Actual | Delta |
|------|----------|--------|-------|
| `test_gltr_dimension_high` | 17.195 | 17.68 | +2.8% |
| `test_gltr_dimension_low` | 4.525 | 4.65 | +2.8% |
| `test_mattr_dimension_excellent` | 12.16 | 12.51 | +2.9% |
| `test_mattr_dimension_poor` | 3.2 | 3.29 | +2.8% |

## Root Cause

The tests calculate expected scores using:
```
expected_score = (dimension_score / 100.0) * dimension_weight
```

Comments in the test file reference Story 2.4.1 weight rescaling (109% → 100%), but the expected values don't match either the original or rescaled weights.

Likely cause: When `figurative_language` dimension fails to load (missing `sentence-transformers`), its weight may be redistributed to other dimensions, causing the ~2.8% increase.

## Steps to Reproduce

```bash
uv run pytest tests/unit/scoring/test_dual_score_calculator.py::TestCalculateDualScoreDimensions::test_gltr_dimension_high -v
```

Output:
```
E assert 17.679499999999997 == 17.195 ± 0.01
E Obtained: 17.679499999999997
E Expected: 17.195 ± 0.01
```

## Proposed Fix

1. Add debug logging to understand runtime weight normalization
2. Document the normalization behavior in architecture docs
3. Update test expectations to match actual behavior with explanatory comments
4. Consider adding a test for the weight normalization logic itself

## Discovery Context

Discovered during Epic 6: Developer Experience Improvements. Verified to fail identically on `main` branch before any Epic 6 changes - this is a pre-existing issue.

## Related Files

- `tests/unit/scoring/test_dual_score_calculator.py` (lines 558-622)
- `src/writescore/scoring/dual_score_calculator.py`
- `src/writescore/dimensions/predictability.py` (weight = 18.1)
- `src/writescore/dimensions/advanced_lexical.py` (weight = 12.8)

## Labels

bug, testing, good first issue

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.