CSV import: delimiter auto-detection returns the first delimiter byte found, not the one that splits the file
- Dominant language
- Rust
- Stars
- 30.5k
- Forks
- 3.2k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 72
Description
**Where**: `rslib/src/import_export/text/csv/metadata.rs`, in `delimiter_from_reader`.
**What happens**: the function reads an 8KB sample and returns the first `Delimiter` variant whose byte appears anywhere in it, iterating in enum order. It never looks at how often the byte occurs or whether it occurs consistently from line to line, so a single occurrence anywhere in the sample decides the delimiter for the whole file. `Colon` is tried before `Comma`.
**Impact**: a comma CSV with no `#separator:` header is detected as colon-delimited whenever any field contains a colon. That covers times (`9:00`), URLs, `parent::child` tag names, and any field beginning `Note:`. The import preview then splits on the wrong character, so fields land in the wrong columns and the notes import wrong.
**Steps to reproduce**:
1. Save this as `test.csv`, with no `#separator:` header:
```
time,note
9:00,wake up
10:30,run
```
2. File > Import, and select the file.
3. The preview splits the rows on `:` instead of `,`.
**Related**: #3853 reported the same symptom and is closed. That report had two causes. The header-line half (a mangled `#separator:Comma,,` line) was fixed by trimming trailing delimiters from `#` comment lines. The content-driven half described in @dae's analysis on that issue is what remains, and it is still reachable today on any file without a valid `#separator:` header.
**Suggested fix**: examine the first several non-empty lines and pick the delimiter that splits them most consistently, meaning the same positive field count on the most lines. That ignores a delimiter appearing only inside field content. Break ties with a priority order that puts genuine delimiters ahead of content-prone ones, so a semicolon file with decimal commas (`1,5;2,7`) stays semicolon. This needs no new dependencies. I have a patch with regression tests in #5166.
Contributor guide
Research direction
Start in rslib/src/import_export/text/csv/metadata.rs at delimiter_from_reader and reproduce the sample from the issue. Inspect the regression tests mentioned in #5166, then verify detection across several non-empty lines, including the comma-with-times and semicolon-with-decimal-commas cases. Done means content delimiters no longer override the delimiter that consistently splits the rows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 62/100