BurntSushi / BurntSushi/rust-csv

Patterns for incrementally parsing chunked data

Open
#176 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2k
Forks
257
PR merge metrics
No merged PRs in 30d

Description

#### What version of the `csv` crate are you using?

name = "csv"
version = "1.1.1"

#### Briefly describe the question, bug or feature request.

Hi, this is not a bug request, but just a question about the usage patterns.
I'm dealing with streaming IO, and I've got my CSV parsing logic in a function
that is repeatedly called with new chunks of data. I've encountered the following problems:

1) Repeatedly constructing new Readers from the input slices "resets" the parsing: all info about headers etc. is lost.
2) On exhausting the end of the input slice, I'd rather revert parsing to last complete record and return the unparsed input to be `chained` as a reader with the next chunk instead of trying to parse the final record and possibly missing the end of it.

So my question is: are the current API's intended to accomodate for this "chunked, incremental" use case, and I'm just failing to find the correct knobs and patterns or I'm I trying to use the csv crate for something it wasn't originally meant for? Would it be possible to add APIs or documentation that made this easier?

#### Include a complete program demonstrating a problem.

https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=0b76f5fd956649247bdc99d471ba8b1b

#### What is the observed behavior of the code above?

```
Parse chunk: Chain { t: [], u: [99, 111, 108, 95, 97, 44, 99, 111, 108, 95, 98, 44, 99, 111, 108, 95, 99, 10, 48, 97, 97, 97, 97, 44, 48, 98, 98, 98, 98, 44, 48, 99, 99, 99, 99, 10, 49, 97, 97, 97, 97, 44, 49, 98, 98, 98, 98, 44, 49, 99, 99] }
Got record: ByteRecord(["0aaaa", "0bbbb", "0cccc"])
Got record: ByteRecord(["1aaaa", "1bbbb", "1cc"])
Done. Unparsed: []
Parse chunk: Chain { t: [], u: [99, 99, 10] }
Done. Unparsed: []
Parse chunk: Chain { t: [], u: [50, 97, 97, 97, 97, 44, 50, 98, 98, 98, 98] }
Done. Unparsed: []
Parse chunk: Chain { t: [], u: [44, 50, 99, 99, 99, 99, 10] }
Done. Unparsed: []
Parse chunk: Chain { t: [], u: [51, 97, 97, 97, 97, 44, 51, 98, 98, 98, 98, 44, 51, 99, 99, 99, 99, 10, 52, 97, 97, 97, 97, 44, 52, 98, 98, 98, 98, 44, 52, 99, 99, 99, 99, 10, 53, 97, 97, 97, 97, 44, 53, 98, 98] }
Got record: ByteRecord(["4aaaa", "4bbbb", "4cccc"])
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error(UnequalLengths { pos: Some(Position { byte: 36, line: 3, record: 2 }), expected_len: 3, len: 2 })', src/libcore/result.rs:1084:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
```
#### What is the expected or desired behavior of the code above?
```
Parse chunk: Chain { t: [], u: [99, 111, 108, 95, 97, 44, 99, 111, 108, 95, 98, 44, 99, 111, 108, 95, 99, 10, 48, 97, 97, 97, 97, 44, 48, 98, 98, 98, 98, 44, 48, 99, 99, 99, 99, 10, 49, 97, 97, 97, 97, 44, 49, 98, 98, 98, 98, 44, 49, 99, 99] }
Got record: ByteRecord(["0aaaa", "0bbbb", "0cccc"])
Done. Unparsed: [49, 97, 97, 97, 97, 44, 49, 98, 98, 98, 98, 44, 49, 99, 99]
Parse chunk: Chain { t: [49, 97, 97, 97, 97, 44, 49, 98, 98, 98, 98, 44, 49, 99, 99], u: [99, 99, 10] }
Got record: ByteRecord(["1aaaa", "1bbbb", "1cccc"])
Done. Unparsed: []
Parse chunk: Chain { t: [], u: [50, 97, 97, 97, 97, 44, 50, 98, 98, 98, 98] }
Done. Unparsed: [50, 97, 97, 97, 97, 44, 50, 98, 98, 98, 98]
Parse chunk: Chain { t: [50, 97, 97, 97, 97, 44, 50, 98, 98, 98, 98], u: [44, 50, 99, 99, 99, 99, 10] }
Got record: ByteRecord(["2aaaa", "2bbbb", "2cccc"])
Done. Unparsed: []
Parse chunk: Chain { t: [], u: [51, 97, 97, 97, 97, 44, 51, 98, 98, 98, 98, 44, 51, 99, 99, 99, 99, 10, 52, 97, 97, 97, 97, 44, 52, 98, 98, 98, 98, 44, 52, 99, 99, 99, 99, 10, 53, 97, 97, 97, 97, 44, 53, 98, 98] }
Got record: ByteRecord(["3aaaa", "3bbbb", "3cccc"])
Got record: ByteRecord(["4aaaa", "4bbbb", "4cccc"])
Done. Unparsed: [53, 97, 97, 97, 97, 44, 53, 98, 98]
Parse chunk: Chain { t: [53, 97, 97, 97, 97, 44, 53, 98, 98], u: [98, 98, 44, 53, 99, 99, 99, 99, 10] }
Got record: ByteRecord(["5aaaa", "5bbbb", "5cccc"])
Done. Unparsed: []
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.