apache / apache/arrow-js

[JS] Decimal-type conversions result in errors/inaccuracies in Arrow JS

Open
#77 7 comments 3 reactions 1 assignee Claimed by @trxcllnt View on GitHub
Type: bug
Dominant language
TypeScript
Stars
112
Forks
23
Avg merge
21h 18m
Merged PRs (30d)
8

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Thanks for all the work on Arrow! Unfortunately, I've run into problems in the JS API around both encoding and decoding of Decimal-typed data.

**Encoding:**

```js
arrow = require('apache-arrow')
arrow.vectorFromArray([1, 12, 34], new arrow.Decimal(3, 18))
```

Results in `TypeError: d.subarray is not a function`

(Incidentally it also seems that the Decimal type constructor uses a different argument order than pyarrow.)

**Decoding:**

Since encoding failed in JS, let's do it in pyarrow:

```py
import pyarrow as pa
v = pa.array([1, 12, 34], type=pa.decimal128(18, 3))
batch = pa.record_batch([v], names=['d'])

sink = pa.BufferOutputStream()
with pa.ipc.new_stream(sink, batch.schema) as writer:
writer.write_batch(batch)
sink.getvalue().hex()
```

and then we can carry the resulting byte string over to JavaScript and try to decode it:

```js
const hex = 'FFFFFFFF780000001000000000000A000C000600050008000A000000000104000C000000080008000000040008000000040000000100000014000000100014000800060007000C00000010001000000000000107100000001C0000000400000000000000010000006400000008000C0004000800080000001200000003000000FFFFFFFF8800000014000000000000000C0016000600050008000C000C0000000003040018000000300000000000000000000A0018000C00040008000A0000003C00000010000000030000000000000000000000020000000000000000000000000000000000000000000000000000003000000000000000000000000100000003000000000000000000000000000000E8030000000000000000000000000000E02E0000000000000000000000000000D0840000000000000000000000000000FFFFFFFF00000000';
const bytes = Uint8Array.from(hex.match(/.{1,2}/g).map(s => parseInt(s, 16)));
const vec = arrow.tableFromIPC(bytes).getChild('d');

vec.get(0) // [1000, 0, 0, 0] <- byte array looks correct
`${vec.get(0)}` // "1000" <- incorrect, should be 1; ignores scale
Number(vec.get(0)) // 2.2136092888451462e+23 <- incorrect, should be 1
+vec.get(0) // TypeError: can't convert BigInt to number <- ???
```

While the basic byte encoding looks OK, all of the built-in conversions appear to be faulty.

Related: apache/arrow-js#100, apache/arrow-js#81
cc: @domoritz

### Component(s)

JavaScript

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.