apache / apache/arrow-js

[JS] Values returned as undefined when arrow file bigger than 2gb

Open
#87 0 comments 0 reactions 0 assignees View on GitHub
Type: bug
Dominant language
TypeScript
Stars
112
Forks
23
Avg merge
21h 18m
Merged PRs (30d)
8

Description

Steps:

 

1. Generate arrow file bigger than 2gb
```java

import pyarrow as pa

nums1 = [42]
nums2 = [42.42]
mil = 1000000

for n in range(1, 140 * mil):
nums1.append(n)
nums2.append(1 / n)

arr1 = pa.array(nums1)
arr2 = pa.array(nums2)

schema = pa.schema([
pa.field('nums1', arr1.type),
pa.field('nums2', arr2.type),
])

with pa.OSFile('arraydata.arrow', 'wb') as sink:
with pa.ipc.new_file(sink, schema=schema) as writer:
batch = pa.record_batch([arr1, arr2], schema=schema)
writer.write(batch)
```
2. Try to read it via the JS SDK
```java

const fs = require("fs");
const { tableFromIPC, RecordBatchReader } = require("apache-arrow");

const filePath = "./arraydata.arrow";

const stream = fs.createReadStream(filePath);
const reader = RecordBatchReader.from(stream);

(async function () {
const table = await tableFromIPC(reader);

console.log("numRows", table.numRows);
console.log("first row", table.get(0).toArray());
})();
```
The code above prints:
```java

numRows 140000000
first row [ undefined, undefined ]
```
`numRows` is correct, but the values are coming out as `{}undefined{`}.

**Reporter**: [Denis Gursky](https://issues.apache.org/jira/browse/ARROW-18007)

**Note**: *This issue was originally created as [ARROW-18007](https://issues.apache.org/jira/browse/ARROW-18007). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.