`convert` date-time results depend on the JVM default time zone
- Dominant language
- Kotlin
- Stars
- 1.1k
- Forks
- 83
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 30
Description
## Summary
`convert`'s date-time conversions resolve zone-dependent steps against `TimeZone.currentSystemDefault()`. The same
operation on the same `DataFrame` produces different values depending on the machine it runs on, and for several
conversions there is no way to pin the zone.
`core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/api/convert.kt:837`:
```kotlin
internal val defaultTimeZone get() = TimeZone.currentSystemDefault()
```
58 entries of the converter table (`impl/api/convert.kt:374`–`:800`) read it. On the public side, 24 overloads in
`api/convert.kt` accept `zone: TimeZone = defaultTimeZone` — but they cover only the receivers `Long`, `Int`,
`kotlin.time.Instant` and `kotlinx.datetime.Instant`, and only the direction towards
`LocalDate`/`LocalTime`/`LocalDateTime`. Everything else takes the ambient zone with no opt-out.
All values below are measured on the current `master` (JDK 21), not derived by hand.
## Why it matters
- **Results are not reproducible across environments.** A pipeline that converts an epoch column to
`LocalDateTime` and then groups, filters, asserts, or persists produces different output on a developer machine
and on CI.
- **The escape hatch is incomplete and its availability depends on how the column was selected**, not on the data
(case 3, case 5).
- **The library already promises the opposite elsewhere.** `docs/StardustDocs/topics/dataSources/ApacheArrow.md`
and `Parquet.md` state that timestamp writing resolves against UTC and never against the default zone, while
`docs/StardustDocs/topics/convert.md` does not mention time zones at all — the string `TimeZone` does not appear
in it.
- **A downstream module already had to work around it.** `dataframe-arrow` reimplemented two of `core`'s
converters privately (`convertToInstantInUtc` / `convertToLocalDateTimeInUtc` in
`dataframe-arrow/.../io/ArrowWriterImpl.kt`) because the same `DataFrame` was writing different Arrow/Parquet
bytes on different machines (#926).
## How it works today
Three properties together mean the zone cannot reach the converter table at all:
| | |
|---|---|
| `internal typealias TypeConverter = (Any) -> Any?` | `impl/api/convert.kt:182` — a converter's only input is the cell value |
| `mutableMapOf, TypeConverter?>` | `impl/api/convert.kt:171` — the converter cache key has no zone, and `ParserOptions` carries none |
| `defaultTimeZone` is a `get()` property | `impl/api/convert.kt:837` — re-read on every call, from inside the per-cell lambda |
Dispatch is reflective on `jvmErasure` in `createConverter` (`impl/api/convert.kt:199`), keyed on the column's
runtime `KType`. Any zone-aware fix has to change `TypeConverter`'s shape or thread a zone through
`convertToTypeImpl` / `getConverter` / `createConverter`.
## Cases
### 1. The same epoch value becomes a different wall-clock reading per machine
```kotlin
val millis = 1_704_110_400_123L // 2024-01-01T12:00:00.123Z
columnOf(millis).convertToLocalDateTime()[0]
```
| Default zone | Result |
|---|---|
| `UTC` | `2024-01-01T12:00:00.123` |
| `Europe/Berlin` | `2024-01-01T13:00:00.123` |
| `Pacific/Kiritimati` | `2024-01-02T02:00:00.123` |
| `America/Los_Angeles` | `2024-01-01T04:00:00.123` |
Note what is *not* broken: `Long → LocalDateTime → Long` does round-trip to `1704110400123` in all four zones,
because both halves read the same ambient zone. The problem is the intermediate value — it is what gets grouped,
compared, asserted and written out — and the round trip only holds while both halves run in the same environment.
For contrast, `Long → Instant` is zone-free and gives `2024-01-01T12:00:00.123Z` in all four zones
(`impl/api/convert.kt:517`, `Instant.fromEpochMilliseconds`), while `Long → LocalDateTime` at `:511` is not. Two
adjacent rows of the same table, two different contracts.
### 2. `LocalDate` has two different meanings of "start of day", nine lines apart
```kotlin
val date = LocalDate(2024, 1, 1)
columnOf(date).convertTo()[0] // zone-free: impl/api/convert.kt:758, atTime(0, 0)
columnOf(date).convertTo()[0] // ambient: impl/api/convert.kt:760, atStartOfDayIn(defaultTimeZone)
columnOf(date).convertToLong()[0] // ambient: impl/api/convert.kt:767
```
| Default zone | `→ LocalDateTime` | `→ Instant` | `→ Long` |
|---|---|---|---|
| `UTC` | `2024-01-01T00:00` | `2024-01-01T00:00:00Z` | `1704067200000` |
| `Europe/Berlin` | `2024-01-01T00:00` | `2023-12-31T23:00:00Z` | `1704063600000` |
| `Pacific/Kiritimati` | `2024-01-01T00:00` | `2023-12-31T10:00:00Z` | `1704016800000` |
| `America/Los_Angeles` | `2024-01-01T00:00` | `2024-01-01T08:00:00Z` | `1704096000000` |
A `LocalDate` of January 1st becomes **December 31st** on two of the four machines. There is no zone parameter on
either of the two ambient conversions.
### 3. Whether you can pin the zone depends on how you selected the column
```kotlin
val df = dataFrameOf("ts")(1_704_110_400_123L)
df.convert("ts").toLocalDateTime() // Convert -> api/convert.kt:2315, no zone
df.convert { ts }.toLocalDateTime(zone) // Convert -> api/convert.kt:2165, zone honoured
```
`convert(vararg columns: String)` returns `Convert` (`api/convert.kt:295`). `Convert` is
covariant, but `Any?` is not a subtype of `Long?`, so the zone-aware overload is inapplicable and resolution falls
to the zone-free `Convert.toLocalDateTime()`. Measured under `Europe/Berlin`, the first line yields
`2024-01-01T13:00:00.123`.
So string-selected and typed-accessor users get different determinism guarantees for the identical column.
### 4. An erased column has no way out at all
For `col: AnyCol` (= `DataColumn<*>`), the only applicable overload is
`DataColumn.convertToLocalDateTime()` (`api/convert.kt:702`), which takes no zone. There is no
`DataColumn<*>.convertToLocalDateTime(zone)` anywhere. Generic code that receives a column cannot be made
deterministic without changing the JVM default zone around the call.
### 5. Conversions with no `zone` parameter anywhere in the public API
```kotlin
val inst = Instant.parse("2024-01-01T23:30:00Z")
columnOf(inst).convertToLocalDate()[0]
```
| Default zone | `→ LocalDate` | `→ LocalTime` |
|---|---|---|
| `UTC` | `2024-01-01` | `23:30` |
| `Europe/Berlin` | `2024-01-02` | `00:30` |
| `Pacific/Kiritimati` | `2024-01-02` | `13:30` |
| `America/Los_Angeles` | `2024-01-01` | `15:30` |
`convertToLocalDate` / `convertToLocalTime` have zone overloads only for `Long` and `Int` receivers — there is no
`DataColumn.convertToLocalDate(zone)`, even though `Instant → LocalDateTime` does get one
(`api/convert.kt:2008`). The whole reverse direction is affected too: `LocalDateTime`/`LocalDate` →
`Instant`/`Long`/`java.time.Instant` (`impl/api/convert.kt:706`, `:713`, `:721`, `:760`, `:767`, `:773`) has no
zone-aware entry point at all. Same for `Byte`/`Short` sources and for every `java.time.*` source.
For these, the only workaround is mutating `java.util.TimeZone.setDefault` around the call.
### 6. DST gaps break the value silently, and only on some machines
```kotlin
val gap = LocalDateTime(2024, 3, 31, 2, 30) // does not exist in Europe/Berlin
columnOf(gap).convertTo()[0]
```
| Default zone | `→ Instant` | back to `LocalDateTime` | round-trip equal |
|---|---|---|---|
| `UTC` | `2024-03-31T02:30:00Z` | `2024-03-31T02:30` | yes |
| `Europe/Berlin` | `2024-03-31T01:30:00Z` | `2024-03-31T03:30` | **no** |
The local time `02:30` does not exist in Berlin on that date, so it is shifted forward by the gap and the round
trip returns `03:30`. The same code on a UTC machine is lossless.
The overlap case (`2024-10-27T02:30`, which happens twice in Berlin) round-trips equal but silently picks the
earlier of the two valid instants (`2024-10-27T00:30:00Z`) — a one-hour ambiguity with no signal to the caller.
### 7. `defaultTimeZone` is read per value, not per column
Because `defaultTimeZone` is a `get()` property read inside the per-cell lambda, a cached converter does not
freeze the zone. A default-zone change during a long-running conversion would produce a column with mixed
interpretations rather than a consistently stale one. This is a property of the mechanism; I did not manage to
produce an observable failure from it, so it is listed as a robustness note rather than a reproducible bug.
## Current test coverage
Worth stating plainly, because it cuts both ways: **there is no value-level test pinning this behaviour in either
direction.** The current behaviour is unprotected, and a change of the default would be invisible to CI.
- `core/src/test/kotlin/.../api/convert.kt` — the test `Instant to LocalDateTime` calls
`df.convert { time }.toLocalDateTime()` on a `Clock.System.now()` column and asserts nothing at all.
- `core/src/test/kotlin/.../io/ParserTests.kt`, `convert to date and time` — the only test of the zone-taking
overloads. It passes `TimeZone.UTC` everywhere and never a second zone, so it would still pass if the `zone`
argument were ignored.
- `samples/src/test/kotlin/.../api/Convert.kt` — the largest cluster of zone-dependent conversions; every
assertion checks type and nullability only, never a value.
- `TimeZone.setDefault` appears nowhere under `core/src/test`. The only helper that runs a block under a fixed
default zone is a private `withDefaultTimeZone` in `dataframe-arrow`'s `ArrowTimestampTzTest`; there is no
shared one in `common-test-utils`.
## Proposed tests
House style is JUnit 4 + kotest matchers, backticked sentence names, one test per claim (`TEST_GUIDELINES.md`).
Natural home: `core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/api/convert.kt` (`ConvertTests`).
A shared helper is needed first — worth promoting into `common-test-utils` rather than copying a third time:
```kotlin
/** Runs [block] with the JVM default time zone set to [zoneId], restoring the previous one afterwards. */
fun withDefaultTimeZone(zoneId: String, block: () -> R): R {
val previous = java.util.TimeZone.getDefault()
java.util.TimeZone.setDefault(java.util.TimeZone.getTimeZone(zoneId))
try {
return block()
} finally {
java.util.TimeZone.setDefault(previous)
}
}
```
**Tests that would fail today only if the default became UTC** (they specify the desired contract, they do not
reproduce a bug):
```kotlin
private val shiftedZones = listOf("UTC", "Europe/Berlin", "Pacific/Kiritimati", "America/Los_Angeles")
@Test
fun `converting a Long to a LocalDateTime does not depend on the default time zone`() {
val millis = 1_704_110_400_123L // 2024-01-01T12:00:00.123Z
shiftedZones.forEach { zone ->
zone.asClue {
withDefaultTimeZone(zone) {
columnOf(millis).convertToLocalDateTime()[0] shouldBe
LocalDateTime(2024, 1, 1, 12, 0, 0, 123_000_000)
}
}
}
}
@Test
fun `converting a LocalDate to an Instant does not depend on the default time zone`() {
shiftedZones.forEach { zone ->
zone.asClue {
withDefaultTimeZone(zone) {
columnOf(LocalDate(2024, 1, 1)).convertTo()[0] shouldBe
Instant.parse("2024-01-01T00:00:00Z")
}
}
}
}
```
**A test that can be added right now, independent of which direction is chosen** — it closes the hole left by
`ParserTests` and passes either way:
```kotlin
@Test
fun `the zone argument is honoured`() {
val col = columnOf(1_704_110_400_123L) // 2024-01-01T12:00:00.123Z
col.convertToLocalDateTime(TimeZone.UTC)[0] shouldBe LocalDateTime(2024, 1, 1, 12, 0, 0, 123_000_000)
col.convertToLocalDateTime(TimeZone.of("Europe/Berlin"))[0] shouldBe
LocalDateTime(2024, 1, 1, 13, 0, 0, 123_000_000)
}
```
Cases 3, 5 and 6 each deserve one test in the same shape.
## Possible directions
Not mutually exclusive, and listed without a recommendation.
**A. Resolve against UTC by default.** Deterministic out of the box, and consistent with what `ApacheArrow.md` and
`Parquet.md` already promise. It is a behavioural breaking change for anyone relying on the ambient zone — though
no existing value-level test asserts the current behaviour, so the change is currently unobservable in CI, which is
an argument both for the change being safe and for it needing new tests either way.
**B. Keep the system default, close the gaps.** Add `zone: TimeZone` wherever it is missing (case 5: `Instant →
LocalDate`/`LocalTime`, the whole local → absolute direction, `Byte`/`Short`, `java.time.*` sources), make it
reachable from an erased receiver (case 4), and document the default in `convert.md` and in the KDoc of the
zone-free overloads. Breaks nothing, but leaves the default non-deterministic.
**C. Require an explicit zone.** Deprecate the zone-less overloads for zone-dependent pairs. Most correct, most
disruptive.
B is a subset of the work needed under any of the three, since the gaps in case 5 leave users with no workaround
except mutating the JVM default.
Orthogonal to all three: `convert.md` should say what the current rule is. It says nothing today.
## Notes
- **The compiler plugin lives in the Kotlin repository**
(`github.com/JetBrains/kotlin/tree/master/plugins/kotlin-dataframe`), not in this one — this repo's
`plugins/kotlin-dataframe` is a disabled, out-of-date legacy copy (see `plugins/AGENTS.md`). Adding or removing a
parameter on any `@Interpretable`-annotated `convert` function therefore needs a matching interpreter change
*there*; changing only the converter internals does not.
- Both `api/convert.kt` and `impl/api/convert.kt` are shaded into `dataframe-compiler-plugin-core`, which the
compiler plugin (and IntelliJ) bundle to run compile-time interpreters — so a change here ships inside the
plugin, and `PluginApiUsages.kt` in that module exercises `convert` directly.
- Related: **#926** (Arrow/Parquet timestamps with a time zone — the downstream half of this),
**#1350** (`kotlinx.datetime.Instant` → `kotlin.time.Instant` migration, which duplicated many of these table
entries).
- Minor, adjacent: the KDoc of the `toLocalDateTime(zone)` overloads says "interpret the … timestamp as a **time**"
— a copy-paste from `toLocalTime` (`api/convert.kt:2158`, `:2180`, `:2202`, `:2224`).
Contributor guide
Research direction
Start with core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/api/convert.kt, especially TypeConverter, getConverter, createConverter, and defaultTimeZone, then inspect the public overloads in core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/api/convert.kt. Read ConvertTests and ParserTests, along with the Arrow writer workaround; done requires an agreed zone contract, coverage across affected conversions, and deterministic results across default zones.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kotlin
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100