Two documents can name the same source and nothing notices: add a `duplicate-source` finding and a `luria merge`
- Dominant language
- Python
- Stars
- 21
- Forks
- 2
- Avg merge
- 6h 33m
- Merged PRs (30d)
- 91
Description
## What happened
Repairing citations in `anthology-of-the-sota`, I corrected `LIT-105`, whose
`arxiv:` had been pointing at an unrelated paper, to `2302.06675`. That is
the right identifier for what the note describes. It is also the identifier
`LIT-090` had been carrying, correctly, the whole time.
So the record now held two notes for one paper:
```
$ grep -l "2302.06675" record/literature.d/*.md
record/literature.d/LIT-090.md
record/literature.d/LIT-105.md
$ luria lint
luria: pending decisions: 10 undecided document(s), oldest 12 days, 0 with unacknowledged references
luria: docs lint clean
```
`docs lint clean`. I only caught it because I happened to grep the corpus
for existing sources while looking for something else. Nothing in the
workflow — `link --fix`, `index`, `lint` — has any reason to look at whether
two documents point at the same thing.
Worth being precise about why this is not a niche case. The two notes had
*different* titles (`Lion: Learning with Implicit Optimization` and
`Symbolic Discovery of Optimization Algorithms`), different authors, and
different statuses — one `Rejected`, one `Proposed`. They disagreed with
each other about a paper the record had already made a curation decision
about, and the disagreement was invisible. This is the same failure DP-2
describes for a field that carries no information, one level up: the
*corpus* was carrying a contradiction and the build was green throughout.
## Why the existing checks miss it
The lint's vocabulary is about references — `retired-citations`,
`unresolved-codes`, `broken-targets`, `remote-drift`. Every one of those
asks "does this pointer resolve?" and each pointer here resolved fine. The
question nobody asks is the converse: *do two documents resolve to the same
place?* `remote-drift` gets closest — it already reads `arxiv:`/`doi:`
through the remotes — but it checks a document against the remote, never
documents against each other.
## What I think is missing
**1. A `duplicate-source` finding.** Group documents in a scheme by their
resolved source — the `arxiv:`, `doi:` or `url:` of the `source` field
group — and report any group with more than one member. It fits the
existing shape exactly: a warning by default, addressable to a person,
promotable through `[luria.lint] fail_on`, acknowledgeable with a directive
for the deliberate case. There is a real deliberate case — a paper filed
once as evidence and once as an object of study, or a report covering two
models the record wants to track separately — so this wants
`duplicate-source-ok:` rather than to be an error.
Cheap version first: exact match on the normalized identifier. That alone
would have caught this. Fuzzy title matching is a different, harder feature
and I would not bundle them.
**2. A `luria merge OLD NEW` command.** The finding tells you there is a
duplicate; resolving one by hand is exactly the kind of mechanical,
multi-file edit the CLI exists to do, and doing it by hand is how references
get missed. The work is:
- rewrite every reference to `OLD` across the repository as `NEW`
- append `OLD` to `NEW`'s `formerly:`, so old spellings in commit messages,
branches and someone's notes still resolve
- retire `OLD` — `Rejected` or `Superseded`, `superseded_by: NEW`, with a
status note — rather than deleting it, per the retire-by-status rule
- leave `OLD`'s body alone, since the two notes' readings may differ and the
losing one is often worth keeping
Nearly all of that machinery already exists. `concretize` already does
"rewrite every reference to this code and record the old spelling in
`formerly:`" — that is its whole job, just triggered by temporary codes
rather than by a merge. `aliases.py` already treats `formerly:` entries as
resolvable. `repair` already knows how to move a note out of `status:` into
`status_note:`/`superseded_by:`. `merge` looks less like new capability than
like composing three things that are each already written.
## The sharper version of the problem
The reason I hit this at all is that a *repair* created the duplicate. The
corpus had ~31 notes whose `arxiv:` resolved to an unrelated paper (a
migration artifact — the identifiers were plausible and wrong). Correcting
them is exactly the operation that collides two documents onto one source,
because the wrong identifiers were, by construction, the thing keeping them
apart.
So a project doing citation repair — which is a normal, expected thing for a
record to need — is walking straight into this with no guard. `luria repair`
already writes mechanical source fixes; if it ever grows "fix a broken
identifier", it should refuse to write one that lands on an identifier
another document already holds, and say which.
## What I did instead
Retired `LIT-105` pointing at `LIT-090`, by hand, and checked the rest of
the corpus for other collisions with a one-off `grep | sort | uniq -d`. That
works once. It is not something a contributor will remember to do, which is
the argument for the finding.
Happy to send a PR for the `duplicate-source` finding if the shape above
looks right — it is small, and I would rather agree on whether it is a
warning class or a `repair` guard before writing it.
Contributor guide
Research direction
Start by reading aliases.py and the existing concretize, repair, and lint paths mentioned in the issue to understand alias resolution, reference rewriting, retirement, and finding directives. Define how duplicate-source groups and deliberate duplicates are represented, then verify the lint finding, merge workflow, and repair guard with focused tests; done means collisions are reported and merges preserve aliases while retiring the old record.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100