kamiazya / kamiazya/web-csv-toolbox
Add example: Processing large CSV files with streaming
- Dominant language
- TypeScript
- Stars
- 21
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
## Description
One of the key features of this library is memory-efficient streaming, but there's no example highlighting this advantage for large file processing.
## Current Status
- Library supports ReadableStream processing
- README mentions streaming but doesn't emphasize memory efficiency
- No example showing real-world large file scenario
## Task
Add a new example section to README.md: **"Processing Large Files Efficiently"**
### What to Include
1. Explanation of why streaming matters for large files
2. Example fetching and processing a large CSV without loading it all into memory
3. Show how to process incrementally (e.g., filtering, transforming)
4. Contrast with array-based approaches
### Example Content
```markdown
### Processing Large Files Efficiently
For large CSV files, use streaming to avoid loading everything into memory:
\`\`\`typescript
import { parse } from 'web-csv-toolbox';
// Fetch and process a large CSV file incrementally
const response = await fetch('https://example.com/large-dataset.csv');
let processedCount = 0;
let filteredCount = 0;
for await (const record of parse(response)) {
processedCount++;
// Process records incrementally
if (record.status === 'active') {
filteredCount++;
// Do something with active records
await saveToDatabase(record);
}
// Show progress every 1000 records
if (processedCount % 1000 === 0) {
console.log(`Processed ${processedCount} records...`);
}
}
console.log(`Filtered ${filteredCount} out of ${processedCount} records`);
\`\`\`
**Memory Efficient**: Records are processed one at a time, so even a 1GB CSV file won't consume 1GB of RAM.
For small datasets where you need all records at once, use `parse.toArray()`:
\`\`\`typescript
const allRecords = await parse.toArray(response);
\`\`\`
```
## Resources
- README.md line 310-313 (existing parse.toArray example)
- Web Streams API benefits
## Acceptance Criteria
- [ ] Add section explaining streaming benefits
- [ ] Show realistic large file processing scenario
- [ ] Contrast with `parse.toArray()` approach
- [ ] Emphasize memory efficiency advantage
---
**Good First Issue**: Showcase the library's key differentiator!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.