mafintosh / mafintosh/csv-parser

Invalid row produced when input data is differently chunked

Open
#159 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
1.5k
Forks
143
PR merge metrics
No merged PRs in 30d

Description

I encountered an issue where csv-parser would randomly append an invalid row to the parsed data, depending on how the data was chunked:

// Normally the last row:
{
  STATIONS_ID: '535',
  MESS_DATUM: '20200309',
  QN_3: '-999',
  FX: '-999',
  FM: '-999',
  QN_4: '1',
  RSK: '6.4',
  RSKF: '4',
  SDK: '4.217',
  SHK_TAG: '0',
  NM: '-999',
  VPM: '7.5',
  PM: '-999',
  TMK: '5.0',
  UPM: '85.71',
  TXK: '7.5',
  TNK: '3.1',
  TGK: '1.3',
  eor: 'eor'
}
// Randomly (differently chunked) we get another row:
{ STATIONS_ID: '' }
  • Operating System: macOS Catalina 10.15.3
  • Node Version: v12.10.0
  • NPM Version: 6.10.3
  • csv-parser: Version: 2.3.2
Expected Behavior

The same result for the same data, no matter how it was chunked.

Actual Behavior

Different results for the same data.

How Do We Reproduce?

https://gist.github.com/hs-rhelm/e95623d9f09621aed96b93b269352049

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the linked reproduction gist against csv-parser 2.3.2 and compare results for differently chunked input. Trace the parser's handling of the final row and chunk boundaries. Done means identical parsed output for the same data, without an extra { STATIONS_ID: '' } row.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.