mafintosh / mafintosh/csv-parser
latin1 supplement separator not working
Open
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 1.5k
- Forks
- 143
- PR merge metrics
- No merged PRs in 30d
Description
- Operating System: macOS Catalina 10.15.7
- Node Version: 14.13.0
- NPM Version: 6.14.8
- csv-parser Version: 3.0.0
Expected Behavior
Using "¬" as a separator in csv file should work. Output should look like:
[ { a: '1', 'b': '2' } ]
Actual Behavior
Parsing seems not correct. Producing something like this:
[ { 'a�b': '1�2' } ]
How Do We Reproduce?
Test file:
a¬b
1¬2
Code:
const csv = require('csv-parser');
const fs = require('fs');
var iconv = require('iconv-lite');
const fileName = ('/tmp/test.csv')
const results = []
fs.createReadStream(fileName)
// .pipe(iconv.decodeStream("utf-8"))
// .pipe(iconv.decodeStream("latin1"))
// .pipe(iconv.encodeStream("utf-8"))
.pipe(csv({
separator: '¬'
}))
.on('data', (row) => {
results.push(row)
})
.on('end', () => {
// console.log(JSON.stringify(results, null, 4))
console.log(results)
console.log('CSV file successfully processed');
});
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the provided Node.js example against the supplied a¬b/1¬2 CSV and compare the expected and actual rows. Trace separator handling in csv({ separator: '¬' }) alongside the shown stream encoding options; done means the output has separate a and b fields with values 1 and 2.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 42/100