apache / apache/arrow-js

[JS] Incorrect Table concat after serialize

Open
#96 0 comments 0 reactions 0 assignees View on GitHub
Type: bug
Dominant language
TypeScript
Stars
112
Forks
23
Avg merge
21h 18m
Merged PRs (30d)
8

Description

After reading an arrow table from disk and concatenating another arrow table, this concatenated table loses it's last concatenated chunk after serializing to disk. 

 

Steps to reproduce: 
```java

// helper that creates a vector to make the below more succinct
function createSomeVectors(){ return [ FloatVector.from(Float32Array.from([1]))] }

// CORRECT
// Create tables T1 + T2. Each contain 1 vector with 1 value
const T1 = Table.new(createSomeVectors(), ['value'])
const T2 = Table.new(createSomeVectors(), ['value'])

// Combine these tables
const combined = T1.concat(T2)

// Serialize and read back this combination (mimic saving and reading from disk)
const combinedAfterSerialization = Table.from([combined.serialize()])

// Print the count (works correctly)
console.log(T1.count(), T2.count(), combined.count(), combinedAfterSerialization.count())
// Result (as expected)= 2, 2, 2, 2

// INCORRECT
// Serialize T1 and read back. (mimic saving and reading from disk)
const T1SerializedAndBack = Table.from([T1.serialize()])

// Combine just read T1SerializedAndBack with T2
const combined2 = T1SerializedAndBack.concat(T2)

// Serialize and read back this combination (mimic saving and reading from disk)
const combinedAfterSerialization2 = Table.from([combined2.serialize()])

// Print the count (works Incorrectly)
console.log(T1SerializedAndBack.count(), T2.count(), combined2.count(), combinedAfterSerialization2.count())
// Result (NOT as expected)= 2, 2, 2, 1 <=!!
```
 

**Reporter**: [geert-jan brits](https://issues.apache.org/jira/browse/ARROW-15086)

**Note**: *This issue was originally created as [ARROW-15086](https://issues.apache.org/jira/browse/ARROW-15086). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.