mafintosh / mafintosh/csv-parser
double quote in data makes parsing exit early without error
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 1.5k
- Forks
- 143
- PR merge metrics
- No merged PRs in 30d
Description
* Operating System: MacOS
* Node Version: 20
* NPM Version: 9.6.6
* csv-parser Version: ^3.0.0
### Expected Behavior
parse a file even if data has a double quote, or at least produce error
### Actual Behavior
silently quits file early
### How Do We Reproduce?
I kept getting less rows streamed than I expected from a file located at https://download.geonames.org/export/dump/admin2Codes.txt
This is a tab delimited file of 45,784 rows
I realized that it was because one of the entries has a double quote
`RU.45.517838 Novotor”yal’skiy Rayon Novotor"yal'skiy Rayon 517838`
, which if I delete it works properly
`RU.45.517838 Novotor[DELETED]yal’skiy Rayon Novotor"yal'skiy Rayon 517838`
```ts
import {Writable} from "node:stream";
import csvParser from "csv-parser";
import {Transform} from "stream";
import https from "https";
const repro = async () => {
let lineCount = 0
return new Promise((resolve, reject) => {
https.get("https://download.geonames.org/export/dump/admin2Codes.txt", (response) => {
response
.pipe(csvParser({separator: "\t", headers: ["id", "name", "nameAscii", "geonameId"]}))
.pipe(new Transform({
objectMode: true,
transform(chunk, encoding, callback) {
lineCount++
this.push(chunk);
callback();
},
}))
.pipe(new Writable({
objectMode: true,
write(chunk, encoding, callback) {
callback();
}
}))
.on('finish', () => {
console.log("total lines should be ~45k", lineCount)
resolve()
})
.on('error', reject)
}).on('error', reject)
})
}
(async () => {
await repro()
})()
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the provided Node.js reproduction, the csvParser({separator: "\t", headers: [...]}) call, and admin2Codes.txt. Investigate why the embedded double quote causes streaming to stop without an error. Done means the complete tab-delimited file is processed or the parser reports an error instead of silently ending.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100