Is there a file size limit for the JS SDK?
- Dominant language
- TypeScript
- Stars
- 112
- Forks
- 23
- Avg merge
- 21h 18m
- Merged PRs (30d)
- 8
Description
```javascript
const fs = require("fs");
const { tableFromIPC, RecordBatchReader } = require("apache-arrow");
const filePath = "./arraydata.arrow";
const stream = fs.createReadStream(filePath);
const reader = RecordBatchReader.from(stream);
(async function () {
const table = await tableFromIPC(reader);
console.log("numRows", table.numRows);
console.log("first row", table.get(0).toArray());
})();
```
The code above prints:
```
numRows 140000000
first row [ undefined, undefined ]
```
`numRows` is correct, but the values are coming out as `undefined`.
`./arraydata.arrow` is generated by the following python code.
```python
import pyarrow as pa
nums1 = [42]
nums2 = [42.42]
mil = 1000000
for n in range(1, 140 * mil):
nums1.append(n)
nums2.append(1 / n)
arr1 = pa.array(nums1)
arr2 = pa.array(nums2)
schema = pa.schema([
pa.field('nums1', arr1.type),
pa.field('nums2', arr2.type),
])
with pa.OSFile('arraydata.arrow', 'wb') as sink:
with pa.ipc.new_file(sink, schema=schema) as writer:
batch = pa.record_batch([arr1, arr2], schema=schema)
writer.write(batch)
```
If I lower the number of rows so that the file gets smaller then the values are coming out just fine. Is there a limit?
Contributor guide
Assessment
This issue has not been assessed yet.