BurntSushi / BurntSushi/rust-csv

Auto-quoting fields containing UTF-8 BOM

Open
#163 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2k
Forks
257
PR merge metrics
No merged PRs in 30d

Description

```
[[package]]
name = "csv"
version = "1.1.1"
```

I'm thinking that, if a record is written that explicitly starts with the UTF-8 BOM (that is `"\u{feff}"`), then `QuoteStyle::Necessary` should trigger quoting of that value.

If the first record/header field in the stream is supposed to contain the BOM as part of the data, but this field is not quoted, then we have ambiguity. Either the BOM pertains to the stream as a whole and should be stripped, or it pertains to the data of the first field and should be preserved.

I agree with the current behavior where the BOM is stripped from the beginning of the stream when reading, and otherwise preserved if present later in the stream. It seems a far more likely scenario that the BOM is part of the UTF-8 encoding of the whole stream, and you don't want the first record polluted by that.

However, when explicitly putting a BOM in a field value I think I'm making my intentions clear that I want it there when writing. If this happens to be a column other than the first, then there is no risk of ambiguity, but if it is the first column, then we end up in the situation above.

It should be the writer's responsibility to avoid this ambiguity. I can avoid it by using `QuoteStyle::Always` but that forces a lot more length onto the output stream. Instead I would like to see either:

* `QuoteStyle::Necessary` invokes quoting if the field begins with a BOM. (<-- my preference)
* `QuoteStyle::Necessary` invokes quoting if the field begins with a BOM *and* this is the first field in the record. (More complicated, less uniform.)

Here is a round-trip test where I explicitly write a BOM at the beginning of the first record, and then try to read it back:

```rust
#[test]
fn test_bom_utf8() {
let orig_record = csv::StringRecord::from(vec!["\u{feff}abc"]);
let mut writer = csv::WriterBuilder::new()
// Switch to QuoteStyle::Always and this is no problem.
.quote_style(csv::QuoteStyle::Necessary)
.from_writer(vec![]);
writer.write_record(orig_record.iter()).unwrap();
let buffer = writer.into_inner().unwrap();
println!("buffer = {:?}", buffer);

let mut reader = csv::ReaderBuilder::new()
.has_headers(false)
.from_reader(&buffer[..]);
let mut result_record = csv::StringRecord::new();
assert!(reader.read_record(&mut result_record).unwrap());
assert_eq!(orig_record, result_record);
}
```

```
running 1 test
buffer = [239, 187, 191, 97, 98, 99, 10]
thread 'test::test_bom_utf8' panicked at 'assertion failed: `(left == right)`
left: `StringRecord(["\u{feff}abc"])`,
right: `StringRecord(["abc"])`', src/main.rs:17:5
```

What do you think?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the provided round-trip test in src/main.rs and trace how QuoteStyle::Necessary handles a field beginning with a UTF-8 BOM. Confirm the reader's existing stream-start behavior, then verify that writing and reading the test record preserves the BOM without requiring QuoteStyle::Always.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.