BurntSushi / BurntSushi/rust-csv
Handling (serialization) of nested containers
- Dominant language
- Rust
- Stars
- 2k
- Forks
- 257
- PR merge metrics
- No merged PRs in 30d
Description
I haven't found any prior discussions on this so I though it might be useful to open an issue for it as it's also a feature request and somewhat of a correctness bug.
The current behavior of this crate is to try to flatten nested containers:
https://docs.rs/csv/1.3.0/csv/struct.Writer.html#rules
> The behavior of serialize is fairly simple:
> 1. Nested containers (tuples, Vecs, structs, etc.) are always flattened (depth-first order).
> 2. If has_headers is true and the type contains field names, then a header row is automatically generated.
>
> However, some container types cannot be serialized, and if has_headers is true, there are some additional restrictions on the types that can be serialized. See below for details.
This design decision is understandable as CSV is pretty limited and simply doesn't support nested containers (vs. JSON, etc.).
However, I do wonder if this is a good default as it interferes with correctness.
Disclaimer: My knowledge of CSV, Rust, and especially this crate is limited but here are my considerations:
The main problem seems to be that there is no official CSV specification. I've used https://en.wikipedia.org/wiki/Comma-separated_values and https://datatracker.ietf.org/doc/html/rfc4180 as references.
(Aside: Perhaps it would also make sense to document to which specification this crate intends to conform to?)
A good rule seems to be the following:
> All records should have the same number of fields, in the same order.
IMO this is a requirement for correctly parsing CSV files (with some exceptions like time series), especially if the first record is a "header" - otherwise there just isn't enough context to understand the structure of the data.
This crate seems to support that rule as well:
https://docs.rs/csv/1.3.0/csv/struct.Reader.html#error-handling
> By default, all records in CSV data must have the same number of fields. If a record is found with a different number of fields than a prior record, then an error is returned. This behavior can be disabled by enabling flexible parsing via the flexible method on ReaderBuilder.
The writer also enforces that rule by default:
https://docs.rs/csv/1.3.0/csv/struct.WriterBuilder.html#method.flexible
> When disabled (which is the default), writing CSV data will return an error if a record is written with a number of fields different from the number of fields written in a previous record.
So the default is to flatten nested containers (like a `Vec` in a `Struct`) and error until `flexible(false)` is used.
It certainly makes sense that `flexible` defaults to `true` but I'd prefer if there was an option to support nested containers without changing the number of fields.
I propose the following ideas (not sure how feasible they are though):
---
### 1. Let the user supply a custom string function to handle the transformation
If possible, this crate would call that function each time it backtracks from such a nested container (optionally with the depth level?). The user could then choose an escaping technique to ensure that the string will represent a single CSV field. This approach is limted as the type information / context is lost but one could at least do simple transformations like quoting and escaping or replacing the delimiters.
Using https://github.com/BurntSushi/rust-csv/issues/254 as an example, the user could use that custom funtion to rewrite `4,5,6,7` (`Vec`) to, e.g., `"4,5,6,7"`, `4;5;6;7`, or `[4,5,6,7]` (like in the desired example). Additional recursion levels could be supported through nested quoting, more delimiters (in that case the function should be called with the depth level), or custom approaches.
Ideally, this would be pretty easy to integrate and flexible enough for most use cases.
### 2. Let the user supply a custom function that handles the nested containers
Similar to 1. but the function would get the raw data and produce the string. The type information would be preserved but it would require reflection, increase the complexitiy, and could be considered out of scope.
### 3. Offer generic techniques to ensure nested containers can be put into a single CSV field
This crate would implement one (or multiple) of the (hopefully pretty universal) custom string functions mentioned in 1. (possibly forcing the introduction of additional constraints).
---
IMO my first proposal could be a decent tradeoff but I might've missed something.
What do you think @BurntSushi?
PS: I'm currently looking into a somewhat "exotic" use case with @ammernico (https://github.com/BurntSushi/rust-csv/issues/254#issuecomment-1822320445) where the types come from an API specification and mainly consist of structures that contain some vectors. This use case makes it difficult to flatten/convert the vectors into a string before passing the data to this CSV crate.
PPS: Huge thanks for this very useful crate and amazing documentation! :)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the documented Writer::serialize rules and the ReaderBuilder and WriterBuilder flexible behavior, then review the related discussion in issue #254. A contribution is not yet well-defined: it would require an agreed nested-container API or serialization policy, with tests demonstrating consistent field counts and the intended behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100