BurntSushi / BurntSushi/rust-csv

Extract a subset of fields, chosen at runtime, by name

Open
#169 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2k
Forks
257
PR merge metrics
No merged PRs in 30d

Description

#### What version of the `csv` crate are you using?

1.1.1

#### Briefly describe your feature request.

I have a program that parses a CSV file with many columns, only some of which are relevant to any given run of the program. Here is a cut-down example CSV file: the real thing has many more rows and also many more columns labeled with uppercase three-character codes.

```csv
addr,id,longitude,latitude,ABW,AFG,AGO,AIA,ALA
2.58.52.152,1716,8.6821267,50.1109221,8138590,4481900,6051510,7167090,1309850
3.17.51.141,1545,-82.9987942,39.9611755,3284140,10847630,10792390,3021880,6873370
3.121.238.252,1546,8.6821267,50.1109221,8138590,4481900,6051510,7167090,1309850
```

On each run of the program, it wants to deserialize `addr`, `longitude`, `latitude`, and _one_ of the columns labeled with a three-character code, into this structure:

```rust
struct HostRecord {
addr: IpAddr,
longitude: f64,
latitude: f64,
distance: f64
}
```

The catch is that _which_ of the three-character-code columns should be deserialized into the `distance` field varies at runtime. There doesn't seem to be any way to make Serde do that. There also doesn't seem to be any way to extract a subset of the fields from a `StringRecord` or `BytesRecord` _by name_. The best I have managed to do is scan the headers record manually for each relevant field, make note of their indices, and then manually extract each field by index, as shown below. It's tedious to write and easy to screw up.

#### Include a complete program demonstrating a problem.

``` rust
struct CSVColumns {
addr: usize,
longitude: usize,
latitude: usize,
distance: usize,
}

/// Error type for select_columns.
#[derive(Debug, Fail)]
#[fail(display="missing columns: {}", _0)]
struct MissingColumnsError(String);

fn select_columns(header: &csv::StringRecord, loc: &str)
-> Result {

let mut addr: Option = None;
let mut longitude: Option = None;
let mut latitude: Option = None;
let mut distance: Option = None;
let mut wanted = 4;

for (index, field) in header.iter().enumerate() {
match field {
"addr" => { addr = Some(index); wanted -= 1; },
"longitude" => { longitude = Some(index); wanted -= 1; },
"latitude" => { latitude = Some(index); wanted -= 1; },
f if f == loc => { distance = Some(index); wanted -= 1; },
_ => {},
}
if wanted == 0 { break }
}

if wanted == 0 {
Ok(CSVColumns {
addr: addr.unwrap(),
longitude: longitude.unwrap(),
latitude: latitude.unwrap(),
distance: distance.unwrap(),
})

} else {
let mut missing: Vec<&str> = Vec::with_capacity(4);
if addr.is_none() { missing.push("addr"); }
if longitude.is_none() { missing.push("longitude"); }
if latitude.is_none() { missing.push("latitude"); }
if distance.is_none() { missing.push(loc); }

Err(MissingColumnsError(missing.join(", ")))
}
}

fn load_hosts(fname: &Path, loc: &str) -> Result, Error> {
use std::str::from_utf8;

let mut fp = File::open(fname)?;
let mut rd = csv::ReaderBuilder::new()
.has_headers(true)
.trim(csv::Trim::All)
.from_reader(fp);
let cols = select_columns(rd.headers()?, loc)?;

// Don't bother doing UTF-8 validation on the columns we're not
// interested in.
let mut row = csv::ByteRecord::new();
let mut v: Vec = Vec::new();
while rd.read_byte_record(&mut row)? {
v.push(HostRecord {
addr: from_utf8(&row[cols.addr])?.parse()?,
longitude: from_utf8(&row[cols.longitude])?.parse()?,
latitude: from_utf8(&row[cols.latitude])?.parse()?,
distance: from_utf8(&row[cols.distance])?.parse()?
});
}
Ok(v)
}
```

#### What is the expected or desired behavior of the code above?

The above code does work, it's just that there should be a better way to write it. Off the top of my head, a plausible "better way" would be a Reader method `find_columns` that tells you the indices for a set of column names, allowing me to dispense with `select_columns` and write something like this instead:

``` rust
fn load_hosts(fname: &Path, loc: &str) -> Result, Error> {
use std::str::from_utf8;

let mut fp = File::open(fname)?;
let mut rd = csv::ReaderBuilder::new()
.has_headers(true)
.trim(csv::Trim::All)
.from_reader(fp);

// find_columns fails if any columns are missing, or produces a
// HashMap<&'a str, usize> mapping column names to indices
let cols = rd.find_columns(&["addr", "latitude", "longitude", loc])?;

// Don't bother doing UTF-8 validation on the columns we're not
// interested in.
let mut row = csv::ByteRecord::new();
let mut v: Vec = Vec::new();
while rd.read_byte_record(&mut row)? {
v.push(HostRecord {
addr: from_utf8(&row[cols["addr"]])?.parse()?,
longitude: from_utf8(&row[cols["longitude"]])?.parse()?,
latitude: from_utf8(&row[cols["latitude"]])?.parse()?,
distance: from_utf8(&row[cols[loc]])?.parse()?
});
}
Ok(v)
}
```

A further improvement would be a way to ask the reader to return only the desired columns from each row, which would both speed up parsing (since only those columns would need to be copied and UTF-8 validated), and enable use of serde again:

``` rust
fn load_hosts(fname: &Path, loc: &str) -> Result, Error> {

use std::str::from_utf8;

let mut fp = File::open(fname)?;

// csv does its own buffering, no need for a BufReader
let mut rd = csv::ReaderBuilder::new()
.has_headers(true)
.trim(csv::Trim::All)
.from_reader(fp);

// find_columns fails if any columns are missing, or produces a
// HashMap<&'a str, usize> mapping column names to indices
let cols = rd.find_columns(&["addr", "latitude", "longitude", loc])?;

let mut row = csv::StringRecord::new();
let mut v: Vec = Vec::new();
while rd.read_columns(&mut row, &cols)? {
v.push(row.deserialize::()?);
}
Ok(v)
}
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.