BurntSushi / BurntSushi/rust-csv
Auto-quoting fields containing UTF-8 BOM
- Dominant language
- Rust
- Stars
- 2k
- Forks
- 257
- PR merge metrics
- No merged PRs in 30d
Description
```
[[package]]
name = "csv"
version = "1.1.1"
```
I'm thinking that, if a record is written that explicitly starts with the UTF-8 BOM (that is `"\u{feff}"`), then `QuoteStyle::Necessary` should trigger quoting of that value.
If the first record/header field in the stream is supposed to contain the BOM as part of the data, but this field is not quoted, then we have ambiguity. Either the BOM pertains to the stream as a whole and should be stripped, or it pertains to the data of the first field and should be preserved.
I agree with the current behavior where the BOM is stripped from the beginning of the stream when reading, and otherwise preserved if present later in the stream. It seems a far more likely scenario that the BOM is part of the UTF-8 encoding of the whole stream, and you don't want the first record polluted by that.
However, when explicitly putting a BOM in a field value I think I'm making my intentions clear that I want it there when writing. If this happens to be a column other than the first, then there is no risk of ambiguity, but if it is the first column, then we end up in the situation above.
It should be the writer's responsibility to avoid this ambiguity. I can avoid it by using `QuoteStyle::Always` but that forces a lot more length onto the output stream. Instead I would like to see either:
* `QuoteStyle::Necessary` invokes quoting if the field begins with a BOM. (<-- my preference)
* `QuoteStyle::Necessary` invokes quoting if the field begins with a BOM *and* this is the first field in the record. (More complicated, less uniform.)
Here is a round-trip test where I explicitly write a BOM at the beginning of the first record, and then try to read it back:
```rust
#[test]
fn test_bom_utf8() {
let orig_record = csv::StringRecord::from(vec!["\u{feff}abc"]);
let mut writer = csv::WriterBuilder::new()
// Switch to QuoteStyle::Always and this is no problem.
.quote_style(csv::QuoteStyle::Necessary)
.from_writer(vec![]);
writer.write_record(orig_record.iter()).unwrap();
let buffer = writer.into_inner().unwrap();
println!("buffer = {:?}", buffer);
let mut reader = csv::ReaderBuilder::new()
.has_headers(false)
.from_reader(&buffer[..]);
let mut result_record = csv::StringRecord::new();
assert!(reader.read_record(&mut result_record).unwrap());
assert_eq!(orig_record, result_record);
}
```
```
running 1 test
buffer = [239, 187, 191, 97, 98, 99, 10]
thread 'test::test_bom_utf8' panicked at 'assertion failed: `(left == right)`
left: `StringRecord(["\u{feff}abc"])`,
right: `StringRecord(["abc"])`', src/main.rs:17:5
```
What do you think?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the provided round-trip test in src/main.rs and trace how QuoteStyle::Necessary handles a field beginning with a UTF-8 BOM. Confirm the reader's existing stream-start behavior, then verify that writing and reading the test record preserves the BOM without requiring QuoteStyle::Always.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100