kamiazya / kamiazya/web-csv-toolbox

Add example: Processing large CSV files with streaming

Open
#511 0 comments 0 reactions 0 assignees View on GitHub
documentation good first issue hacktoberfest
Dominant language
TypeScript
Stars
21
Forks
10
PR merge metrics
No merged PRs in 30d

Description

## Description
One of the key features of this library is memory-efficient streaming, but there's no example highlighting this advantage for large file processing.

## Current Status
- Library supports ReadableStream processing
- README mentions streaming but doesn't emphasize memory efficiency
- No example showing real-world large file scenario

## Task
Add a new example section to README.md: **"Processing Large Files Efficiently"**

### What to Include
1. Explanation of why streaming matters for large files
2. Example fetching and processing a large CSV without loading it all into memory
3. Show how to process incrementally (e.g., filtering, transforming)
4. Contrast with array-based approaches

### Example Content
```markdown
### Processing Large Files Efficiently

For large CSV files, use streaming to avoid loading everything into memory:

\`\`\`typescript
import { parse } from 'web-csv-toolbox';

// Fetch and process a large CSV file incrementally
const response = await fetch('https://example.com/large-dataset.csv');

let processedCount = 0;
let filteredCount = 0;

for await (const record of parse(response)) {
processedCount++;

// Process records incrementally
if (record.status === 'active') {
filteredCount++;
// Do something with active records
await saveToDatabase(record);
}

// Show progress every 1000 records
if (processedCount % 1000 === 0) {
console.log(`Processed ${processedCount} records...`);
}
}

console.log(`Filtered ${filteredCount} out of ${processedCount} records`);
\`\`\`

**Memory Efficient**: Records are processed one at a time, so even a 1GB CSV file won't consume 1GB of RAM.

For small datasets where you need all records at once, use `parse.toArray()`:

\`\`\`typescript
const allRecords = await parse.toArray(response);
\`\`\`
```

## Resources
- README.md line 310-313 (existing parse.toArray example)
- Web Streams API benefits

## Acceptance Criteria
- [ ] Add section explaining streaming benefits
- [ ] Show realistic large file processing scenario
- [ ] Contrast with `parse.toArray()` approach
- [ ] Emphasize memory efficiency advantage

---
**Good First Issue**: Showcase the library's key differentiator!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.