adaltas / adaltas/node-csv

Implement parse data from AsyncIterator

Open
#447 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
JavaScript
Stars
4.3k
Forks
299
Avg merge
16h 19m
Merged PRs (30d)
1

Description

Hello, it's me again )

The streams work well, but I decided to optimize them even more by removing all unnecessary abstractions, leaving only the native JS (AsyncIterator) and the result is worth it.

I propose to add a new simple interface for parsing data from an asynchronous iterator (generator), it is a great performance and a very simple implementation.

User land example:

import { parse } from 'csv-parse/iterator';

async function* iterator() {
  try {
    yield Buffer.from('A,B,C\n');
  } catch (error) {
    console.error(error);
  }
}

for await (const records of parse(iterator()))
  console.log(records);

Lib implement:

async function* parse(iterator) {
  let result = null;
  const setResult = records => {
    result = records;
  };

  for await (const chunk of iterator) {
    const error = api.parse(chunk, false, setResult);

    if (error) {
      await iterator.throw(error);
    } else if (result) {
      yield result;
      result = null;
    }
  }

  // Flush
  const error = api.parse(undefined, true, setResult);

  if (error) {
    await iterator.throw(error);
  } else if (result) {
    return result;
  }
}

Asynchronous iterators are great for this task, they work in all JavaScript environments, consume less memory and CPU compared to any stream implementation.

In fact, there are a lot of sources in the form of asynchronous iterators, all streams provide an interface for asynchronous iterators, here is an example of fetch:

import { parse } from 'csv-parse/iterator';

const response = await fetch('file.csv');

for await (const record of parse(response[Symbol.asyncIterator]())) {
   console.log(record);
}

According to my local measurements, async iterators are 20% faster than streams, and at the user level, writing a generator function is much faster than coding using stream interfaces.

If you like it, I can do a PR

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the proposed csv-parse/iterator entry point and the existing api.parse callback contract shown in the issue. Verify iteration, flushing, yielded records, and iterator.throw behavior with asynchronous inputs; done means the new interface handles the demonstrated generator and fetch-style async iterators correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.