max-mapper / max-mapper/dat-wise-stripe-82
parsing csv is slow
Open
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 1
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
to reproduce:
download this file (at least the first few MBs of it, its a ~9GB file):
http://irsadist.ipac.caltech.edu/wise-allwise/wise-allwise-cat-part26.gz
pipe it into the parser command:
npm i
cat wise-allwise-cat-part26.gz | npm test
it's only parsing like 1500 rows/s for me :(
cc @mafintosh
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by downloading a few megabytes of the linked catalog, running npm i, and reproducing the npm test pipeline to confirm the reported rate. Inspect the parser entry point and benchmark where time is spent; done means improving the documented roughly 1,500 rows/s throughput, though the issue provides no target rate or file paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100