adaltas / adaltas/node-csv

Cannot parse from a stream in the browser

Open
#443 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
4.3k
Forks
299
Avg merge
16h 19m
Merged PRs (30d)
1

Description

Describe the bug

I want to allow a user to upload a gzipped csv file and parse it in a stream (using pipeThrough / pipeTo). This works in Node, but not in the browser, where I get:

file-upload.html:52 Uncaught (in promise) TypeError: Failed to execute 'pipeTo' on 'ReadableStream': parameter 1 is not of type 'WritableStream'.
    at HTMLInputElement.<anonymous> (file-upload.html:52:18)

Line 52 is the pipeTo line from the example below, so it seems to think that the parse object is not a WritableStream.

To Reproduce

Dump the code below into a file, open it in a browser and then try uploading a .tsv.gz file. I tested in Chrome and Firefox. The same code works in Node, although you have to swap pipeThrough and pipeTo for pipe. I'll admit that I find the differences between Node and the browser for streaming a bit confusing so I may have made some obvious error here.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>File Upload</title>
</head>
<body>
    <div class="upload-container">
        <form class="upload-form">
            <input type="file" id="fileInput" name="file">
        </form>
    </div>

    <script type="module">
        import { parse } from 'https://cdn.jsdelivr.net/npm/csv-parse@5.5.6/+esm'

        document.querySelector('#fileInput').addEventListener('change', (e) => {

            e.preventDefault();
            const file = document.getElementById('fileInput').files[0];

            // turn the file into a stream
            const fileStream = file.stream()

            // create a gunzip transform stream
            const unzipper = new DecompressionStream('gzip')    

            const records = [];

            // create a csv parser writeable stream
            const parser = parse({
              delimiter: '\t'
            });            
 
            parser.on('readable', function(){
              let record;
              while ((record = parser.read()) !== null) {
                records.push(record);
              }
            });
            parser.on('error', function(err){
              console.error(err.message);
            });
            parser.on('end', function(){
                console.log(records)
            })

            console.log("piping!")
            // pipe the file stream through the gunzip stream and into the csv parse stream
            fileStream
                .pipeThrough(unzipper)
                .pipeTo(parser)
        });
    </script>
</body>
</html>

** Additional Context **

I wonder if this issue is related, although I'm not trying to parse the complex CSVs listed in that issue.

I also have a similar issue raised for Papaparse.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the browser example in file-upload.html, especially line 52 where the file stream is piped into the parser, and compare the parser export with the browser WritableStream contract. Reproduce the upload in Chrome or Firefox and verify that a gzipped TSV reaches the parser and emits the expected records without the pipeTo type error.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
stream-processing, web-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.