apache / apache/arrow-js

[JS] Table.fromStruct() silently truncates vectors to the first chunk

Open
#104 2 comments 0 reactions 0 assignees View on GitHub
Type: bug
Dominant language
TypeScript
Stars
112
Forks
23
Avg merge
21h 18m
Merged PRs (30d)
8

Description

Table.fromStruct() only uses the first chunk from the input vector.
```javascript

import { Bool, Field, Int32, Struct, Table, Vector } from "apache-arrow";

const myStruct = new Struct([
Field.new({ name: "over", type: new Int32() }),
Field.new({ name: "out", type: new Bool() })
]);

const data = [];
for(let i=0;i<1500;i++) {
data.push({ over:i, out:i%2 === 0 });

// create a vector with two chunks
const victor = Vector.from({
type: myStruct,
/*highWaterMark: Infinity,*/
values: data
});
console.log(victor.length); // 1500

const table = Table.fromStruct(victor);
console.log(table.length); // 1000

```
 The workaround is to set highWaterMark to Infinity

 

Table.new() works as expected
```javascript

const int32Array = new Int32Array(1500);for(let i=0;i<1500;i++) int32Array[i] = i;
const intVector = Vector.from({ type: new Int32(), values: int32Array});
console.log(intVector.length); // 1500

 const intTable = Table.new({ intColumn:intVector });
console.log(intTable.length); // 1500

```
 

The origin seems to be in Chunked.data() but I don't understand the code enough to propose a fix.

**Reporter**: [David Saslawsky](https://issues.apache.org/jira/browse/ARROW-10450)

**Note**: *This issue was originally created as [ARROW-10450](https://issues.apache.org/jira/browse/ARROW-10450). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.