apache / apache/arrow-js

Is there a file size limit for the JS SDK?

Open
#139 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
112
Forks
23
Avg merge
21h 18m
Merged PRs (30d)
8

Description

```javascript
const fs = require("fs");
const { tableFromIPC, RecordBatchReader } = require("apache-arrow");

const filePath = "./arraydata.arrow";

const stream = fs.createReadStream(filePath);
const reader = RecordBatchReader.from(stream);

(async function () {
const table = await tableFromIPC(reader);

console.log("numRows", table.numRows);
console.log("first row", table.get(0).toArray());
})();

```

The code above prints:
```
numRows 140000000
first row [ undefined, undefined ]
```

`numRows` is correct, but the values are coming out as `undefined`.

`./arraydata.arrow` is generated by the following python code.

```python
import pyarrow as pa

nums1 = [42]
nums2 = [42.42]
mil = 1000000

for n in range(1, 140 * mil):
nums1.append(n)
nums2.append(1 / n)

arr1 = pa.array(nums1)
arr2 = pa.array(nums2)

schema = pa.schema([
pa.field('nums1', arr1.type),
pa.field('nums2', arr2.type),
])

with pa.OSFile('arraydata.arrow', 'wb') as sink:
with pa.ipc.new_file(sink, schema=schema) as writer:
batch = pa.record_batch([arr1, arr2], schema=schema)
writer.write(batch)
```

If I lower the number of rows so that the file gets smaller then the values are coming out just fine. Is there a limit?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.