BurntSushi / BurntSushi/rust-csv
Patterns for incrementally parsing chunked data
- Dominant language
- Rust
- Stars
- 2k
- Forks
- 257
- PR merge metrics
- No merged PRs in 30d
Description
#### What version of the `csv` crate are you using?
name = "csv"
version = "1.1.1"
#### Briefly describe the question, bug or feature request.
Hi, this is not a bug request, but just a question about the usage patterns.
I'm dealing with streaming IO, and I've got my CSV parsing logic in a function
that is repeatedly called with new chunks of data. I've encountered the following problems:
1) Repeatedly constructing new Readers from the input slices "resets" the parsing: all info about headers etc. is lost.
2) On exhausting the end of the input slice, I'd rather revert parsing to last complete record and return the unparsed input to be `chained` as a reader with the next chunk instead of trying to parse the final record and possibly missing the end of it.
So my question is: are the current API's intended to accomodate for this "chunked, incremental" use case, and I'm just failing to find the correct knobs and patterns or I'm I trying to use the csv crate for something it wasn't originally meant for? Would it be possible to add APIs or documentation that made this easier?
#### Include a complete program demonstrating a problem.
https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=0b76f5fd956649247bdc99d471ba8b1b
#### What is the observed behavior of the code above?
```
Parse chunk: Chain { t: [], u: [99, 111, 108, 95, 97, 44, 99, 111, 108, 95, 98, 44, 99, 111, 108, 95, 99, 10, 48, 97, 97, 97, 97, 44, 48, 98, 98, 98, 98, 44, 48, 99, 99, 99, 99, 10, 49, 97, 97, 97, 97, 44, 49, 98, 98, 98, 98, 44, 49, 99, 99] }
Got record: ByteRecord(["0aaaa", "0bbbb", "0cccc"])
Got record: ByteRecord(["1aaaa", "1bbbb", "1cc"])
Done. Unparsed: []
Parse chunk: Chain { t: [], u: [99, 99, 10] }
Done. Unparsed: []
Parse chunk: Chain { t: [], u: [50, 97, 97, 97, 97, 44, 50, 98, 98, 98, 98] }
Done. Unparsed: []
Parse chunk: Chain { t: [], u: [44, 50, 99, 99, 99, 99, 10] }
Done. Unparsed: []
Parse chunk: Chain { t: [], u: [51, 97, 97, 97, 97, 44, 51, 98, 98, 98, 98, 44, 51, 99, 99, 99, 99, 10, 52, 97, 97, 97, 97, 44, 52, 98, 98, 98, 98, 44, 52, 99, 99, 99, 99, 10, 53, 97, 97, 97, 97, 44, 53, 98, 98] }
Got record: ByteRecord(["4aaaa", "4bbbb", "4cccc"])
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error(UnequalLengths { pos: Some(Position { byte: 36, line: 3, record: 2 }), expected_len: 3, len: 2 })', src/libcore/result.rs:1084:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
```
#### What is the expected or desired behavior of the code above?
```
Parse chunk: Chain { t: [], u: [99, 111, 108, 95, 97, 44, 99, 111, 108, 95, 98, 44, 99, 111, 108, 95, 99, 10, 48, 97, 97, 97, 97, 44, 48, 98, 98, 98, 98, 44, 48, 99, 99, 99, 99, 10, 49, 97, 97, 97, 97, 44, 49, 98, 98, 98, 98, 44, 49, 99, 99] }
Got record: ByteRecord(["0aaaa", "0bbbb", "0cccc"])
Done. Unparsed: [49, 97, 97, 97, 97, 44, 49, 98, 98, 98, 98, 44, 49, 99, 99]
Parse chunk: Chain { t: [49, 97, 97, 97, 97, 44, 49, 98, 98, 98, 98, 44, 49, 99, 99], u: [99, 99, 10] }
Got record: ByteRecord(["1aaaa", "1bbbb", "1cccc"])
Done. Unparsed: []
Parse chunk: Chain { t: [], u: [50, 97, 97, 97, 97, 44, 50, 98, 98, 98, 98] }
Done. Unparsed: [50, 97, 97, 97, 97, 44, 50, 98, 98, 98, 98]
Parse chunk: Chain { t: [50, 97, 97, 97, 97, 44, 50, 98, 98, 98, 98], u: [44, 50, 99, 99, 99, 99, 10] }
Got record: ByteRecord(["2aaaa", "2bbbb", "2cccc"])
Done. Unparsed: []
Parse chunk: Chain { t: [], u: [51, 97, 97, 97, 97, 44, 51, 98, 98, 98, 98, 44, 51, 99, 99, 99, 99, 10, 52, 97, 97, 97, 97, 44, 52, 98, 98, 98, 98, 44, 52, 99, 99, 99, 99, 10, 53, 97, 97, 97, 97, 44, 53, 98, 98] }
Got record: ByteRecord(["3aaaa", "3bbbb", "3cccc"])
Got record: ByteRecord(["4aaaa", "4bbbb", "4cccc"])
Done. Unparsed: [53, 97, 97, 97, 97, 44, 53, 98, 98]
Parse chunk: Chain { t: [53, 97, 97, 97, 97, 44, 53, 98, 98], u: [98, 98, 44, 53, 99, 99, 99, 99, 10] }
Got record: ByteRecord(["5aaaa", "5bbbb", "5cccc"])
Done. Unparsed: []
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.