mafintosh / mafintosh/csv-parser

Option 'reduceValues' to give users control over the object constructed for eah row

Open
#174 0 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
1.5k
Forks
143
PR merge metrics
No merged PRs in 30d

Description

## Feature Proposal

Provide a new option to the options object that allows users to choose how to modify the output object based on each cell in the row. Documentation provided below:

### reduceValues

Type: `Function`

A function that can be used to modify the object that is emitted by the stream, similar to a [reducer function](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/reduce). The return value will replace the existing object used accumulating columns in the row. If `null` is returned, the rest of the row is skipped (similar to mapHeaders).

```javascript
csv({
reduceValues: ({ memo, header, index, value }) => memo[header] === '' ? (memo[header] = value, memo) : memo
})
```

### Parameters

**memo** _Object or any_ The current object representing the values in a row
**header** _String_ The current column header.
**index** _Number_ The current column index.
**value** _String or any_ The current column value (or content).

If both `mapValues` and `reduceValues` functions are provided, `mapValues` is run first, and the output is provided in the **value** parameter of the `reduceValues` function.

## Feature Use Case

This feature gives the developer more control over constructing the object that is passed through the stream, or choosing to skip a row based on the values encountered for a row. Some specific example use case:

- If multiple columns have duplicate-named headers, instead of automatically taking the last one, a user could choose to take the one that does not have an empty value, or take the max of the values.
- Skip records (rows) if the first column is blank
- If headers are named like a JS object path, eg. `first[0].inner`, user can handle logic of constructing nested objects
- Rehydrating a row with a particular type of object, ie. on `index === 0`, instantiate a `new RowObject()`

The goal of this proposal is to allow developers more freedom over the deserialization without increasing maintenance burden for the maintainers.

** Why not just make the user transform the object downstream from the CSV parser. After all, this is only a CSV parser!!! **

Moving the object reducing logic upstream can help maintain the fast speed this library is known for, as well as allow users to build their own workaround for issues like #150. If a user has more control over skipping a row, or can stop deserializing the rest of the row, there are performance advantages, especially in very wide data sets or ones with lots of rows to be skipped. In addition, some behaviors, like when there are duplicate header names, cannot be addressed downstream.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the existing mapValues and mapHeaders option handling and the row-construction path. Compare those behaviors with the proposed reduceValues parameters, return values, ordering, and duplicate-header use cases. Done means the documented option behavior is implemented and its interaction with mapValues and row skipping is verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.