ankitects / ankitects/anki

CSV import: delimiter auto-detection returns the first delimiter byte found, not the one that splits the file

Open
#5,253 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
30.5k
Forks
3.2k
Avg merge
2d 8h
Merged PRs (30d)
72

Description

**Where**: `rslib/src/import_export/text/csv/metadata.rs`, in `delimiter_from_reader`.

**What happens**: the function reads an 8KB sample and returns the first `Delimiter` variant whose byte appears anywhere in it, iterating in enum order. It never looks at how often the byte occurs or whether it occurs consistently from line to line, so a single occurrence anywhere in the sample decides the delimiter for the whole file. `Colon` is tried before `Comma`.

**Impact**: a comma CSV with no `#separator:` header is detected as colon-delimited whenever any field contains a colon. That covers times (`9:00`), URLs, `parent::child` tag names, and any field beginning `Note:`. The import preview then splits on the wrong character, so fields land in the wrong columns and the notes import wrong.

**Steps to reproduce**:

1. Save this as `test.csv`, with no `#separator:` header:
```
time,note
9:00,wake up
10:30,run
```
2. File > Import, and select the file.
3. The preview splits the rows on `:` instead of `,`.

**Related**: #3853 reported the same symptom and is closed. That report had two causes. The header-line half (a mangled `#separator:Comma,,` line) was fixed by trimming trailing delimiters from `#` comment lines. The content-driven half described in @dae's analysis on that issue is what remains, and it is still reachable today on any file without a valid `#separator:` header.

**Suggested fix**: examine the first several non-empty lines and pick the delimiter that splits them most consistently, meaning the same positive field count on the most lines. That ignores a delimiter appearing only inside field content. Break ties with a priority order that puts genuine delimiters ahead of content-prone ones, so a semicolon file with decimal commas (`1,5;2,7`) stays semicolon. This needs no new dependencies. I have a patch with regression tests in #5166.

Contributor guide

Open the contributing guide

Research direction

Start in rslib/src/import_export/text/csv/metadata.rs at delimiter_from_reader and reproduce the sample from the issue. Inspect the regression tests mentioned in #5166, then verify detection across several non-empty lines, including the comma-with-times and semicolon-with-decimal-commas cases. Done means content delimiters no longer override the delimiter that consistently splits the rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.