ankitects / ankitects/anki

Duplicate note validation does not ignore outer whitespace symmetrically

Open
#5,184 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
30.5k
Forks
3.2k
Avg merge
2d 8h
Merged PRs (30d)
60

Description

### Problem

Duplicate note validation in the add-note editor compares the first field after
HTML stripping, but leading and trailing whitespace remain part of both the
comparison value and its checksum.

As a result, visually equivalent first fields may not be reported as duplicates.

### Steps to reproduce

#### Direction 1

1. Create a Basic note whose first field is `apple`.
2. Start creating another note of the same note type.
3. Enter `apple ` in the first field, with one trailing space.

#### Direction 2

1. Create a Basic note whose first field is ` apple `, with surrounding spaces.
2. Start creating another note of the same note type.
3. Enter `apple` in the first field.

### Actual behavior

The candidate may not be reported as a duplicate because surrounding whitespace
changes the first-field checksum.

### Expected behavior

For duplicate validation only, first fields that differ solely in leading or
trailing whitespace should be treated as duplicates in both directions:

- existing `apple`, candidate `apple ` → duplicate
- existing ` apple `, candidate `apple` → duplicate

Internal whitespace should remain meaningful:

- existing `New York`, candidate `NewYork` → not duplicate
- existing `New York`, candidate ` New York ` → duplicate

The stored field content should remain unchanged. This proposal is not intended
to automatically trim note fields when saving them.

### Current implementation constraint

The stored checksum is calculated from the HTML-stripped but untrimmed first
field.

Duplicate validation then calculates the candidate checksum and queries notes by
checksum and note type before comparing the full stripped first field.

Because of this, a small candidate-side trim would fix only one direction:

- existing `apple`, candidate ` apple ` → duplicate

but not necessarily the reverse direction:

- existing ` apple `, candidate `apple` → duplicate

Trimming both values after the checksum lookup would not solve this, because an
existing note stored under a whitespace-bearing checksum would not be returned by
a lookup for the checksum of the trimmed value.

The same checksum behavior is also used in note saving, text import duplicate
matching, and duplicate searches, so changing only one path may create
inconsistent behavior.

### Possible implementation scopes

There appear to be several possible approaches:

1. Trim only the candidate during duplicate validation.

This would be a small change, but the behavior would remain asymmetric for
existing notes already stored with surrounding whitespace.

2. Normalize the checksum input consistently.

The stored field text would remain unchanged, but the checksum would be based
on the HTML-stripped and outer-trimmed first field.

This would provide efficient symmetric lookups, but may require coordinated
changes in note saving, editor validation, text imports, duplicate searches,
and possibly a checksum rebuild or migration for existing collections.

3. Add a fallback comparison for notes of the same note type.

The current checksum lookup could remain the fast path. If it does not find a
normalized match, Anki could compare the trimmed, HTML-stripped first fields of
notes with the same note type.

This would work with existing collections, but may have performance costs
during live editor validation.

4. Preserve the current exact-whitespace behavior.

### Maintainer guidance requested

Before preparing a pull request, I would like to confirm which behavior and
implementation scope the maintainers prefer.

Would you prefer:

- the smallest candidate-side fix, despite its asymmetry;
- a symmetric checksum-normalization change with any required migration or
rebuild;
- a fallback comparison approach, with performance benchmarks;
- or no change to the current exact-whitespace behavior?

Once the preferred approach is confirmed, I will implement only that solution in
a focused pull request with Rust tests.

Contributor guide

Open the contributing guide

Research direction

No files or test paths are named. First trace the checksum lookup and comparison used by editor validation, then inspect the related note-saving, text-import matching, and duplicate-search paths; wait for maintainer guidance on the intended scope before changing behavior. Done means the chosen approach has Rust tests covering both whitespace directions without changing stored field content.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.