[JS] pre-epoch Timestamps are off by a fixed amount
- Dominant language
- TypeScript
- Stars
- 112
- Forks
- 23
- Avg merge
- 21h 18m
- Merged PRs (30d)
- 8
Description
### Describe the bug, including details regarding any error messages, version, and platform.
While constructing a `Vector` of `Timestamp` values, any pre-epoch values end up offset by approximately `2^32 * 10^(3n)`, where `n` depends on the precision.
I've written a script for minimal repro using the latest version of the apache-arrow NPM package (12.0.0).
```javascript
// test.mjs
import {
Date_,
makeBuilder,
Timestamp,
TimeUnit,
DateUnit,
} from "apache-arrow";
/**
* @param {Date} value
* @param {TimeUnit} precision
* @returns {number}
*/
const testTimestamp = (value, precision) => {
const columnBuilder = makeBuilder({ type: new Timestamp(precision) });
columnBuilder.append(value);
const vec = columnBuilder.finish().toVector();
const valueInVec = Array.from(vec)[0];
return +value - +valueInVec;
};
console.log("Testing TIMESTAMP\n");
[
["2000-01-01", TimeUnit.NANOSECOND],
["2000-01-01", TimeUnit.MICROSECOND],
["2000-01-01", TimeUnit.MILLISECOND],
["2000-01-01", TimeUnit.SECOND],
[],
["1900-01-01", TimeUnit.NANOSECOND],
["1900-01-01", TimeUnit.MICROSECOND],
["1900-01-01", TimeUnit.MILLISECOND],
["1900-01-01", TimeUnit.SECOND],
[],
["1969-12-31 23:59:59Z", TimeUnit.NANOSECOND],
["1969-01-01", TimeUnit.NANOSECOND],
["1900-01-01", TimeUnit.NANOSECOND],
["1800-01-01", TimeUnit.NANOSECOND],
["1700-01-01", TimeUnit.NANOSECOND],
].forEach(([dateStr, precision]) => {
if (!dateStr) {
console.log();
return;
}
const outcome = testTimestamp(new Date(dateStr), precision);
const label = `${dateStr} w/ ${TimeUnit[precision]}`;
console.log(`${label.padEnd(40)} => ${outcome}`);
});
console.log("\n\nTesting DATE\n");
/**
* @param {Date} value
* @returns {Date}
*/
const testDate = (value, precision) => {
const columnBuilder = makeBuilder({ type: new Date_(DateUnit.DAY) });
columnBuilder.append(value);
const vec = columnBuilder.finish().toVector();
const valueInVec = Array.from(vec)[0];
return valueInVec;
};
[
["1969-12-15 01:00:00Z", "rounded up"],
["1969-12-31 01:00:00Z", "rounded up"],
["1970-01-01 00:00:00Z", "rounded down"],
["1970-01-01 23:00:00Z", "rounded down"],
["1970-01-15 01:00:00Z", "rounded down"],
].forEach(([dateStr, label]) => {
const date = new Date(dateStr);
const outcome = testDate(date);
console.log(`${date.toISOString()} => ${outcome.toISOString()} (${label})`);
});
```
Here's the output:
```
Testing TIMESTAMP
2000-01-01 w/ NANOSECOND => 0
2000-01-01 w/ MICROSECOND => 0
2000-01-01 w/ MILLISECOND => 0
2000-01-01 w/ SECOND => 0
1900-01-01 w/ NANOSECOND => -4294.96728515625
1900-01-01 w/ MICROSECOND => -4294967.2958984375
1900-01-01 w/ MILLISECOND => -4294967296
1900-01-01 w/ SECOND => -4294967296000
1969-12-31 23:59:59Z w/ NANOSECOND => -4294.967296
1969-01-01 w/ NANOSECOND => -4294.967296600342
1900-01-01 w/ NANOSECOND => -4294.96728515625
1800-01-01 w/ NANOSECOND => -4294.9677734375
1700-01-01 w/ NANOSECOND => -4294.966796875
Testing DATE
1969-12-15T01:00:00.000Z => 1969-12-16T00:00:00.000Z (rounded up)
1969-12-31T01:00:00.000Z => 1970-01-01T00:00:00.000Z (rounded up)
1970-01-01T00:00:00.000Z => 1970-01-01T00:00:00.000Z (rounded down)
1970-01-01T23:00:00.000Z => 1970-01-01T00:00:00.000Z (rounded down)
1970-01-15T01:00:00.000Z => 1970-01-15T00:00:00.000Z (rounded down)
```
As you can see, for the `Timestamp` type, values post-epoch pass through unscathed, but values pre-epoch end up off by a fixed increment of:
- approximately `2^32 / 1,000,000` for `NANOSECOND` precision
- approximately `2^32 / 1,000` for `MICROSECOND` precision
- `2^32` for `MILLISECOND` precision
- `2^32 * 1,000` for `SECOND` precision
Note: I say "fixed", but the amount actually varies slightly when it's a float (for `NANOSECOND` and `MICROSECOND` precisions), as you can see in the last block of tests for `Timestamp` -- it varies a bit, seemingly varying more as the value gets further from the epoch.
It seems like the Arrow `Timestamp` type is a 64-bit int of {precision} since the epoch, but is represented as two 32-bit ints in JS. Something is getting messed up for pre-epoch timestamps, which have a negative number for their internal representation. I'm guessing it has to do with 32bit arithmetic and/or float precision issues.
Additionally, for the `Date_` type, values post-epoch seem to "round down" to the nearest day properly, but values pre-epoch seem to "round up" to the next day. I'm guessing that this is a related issue.
### Component(s)
JavaScript
Contributor guide
Assessment
This issue has not been assessed yet.