apache / apache/parquet-java

Accept local timestamps annotated with the legacy timestamp types

Offen
#1,562 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Component: Java Component: Parquet Priority: Major Type: task
Vorherrschende Sprache
Java
Sterne
3.1k
Forks
1.6k
Ø Merge
3 T. 12 Std.
Gemergte PRs (30 T.)
33

Beschreibung

The rules for TIMESTAMP forward-compatibility were created based on the assumption that TIMESTAMP_MILLIS and TIMESTAMP_MICROS have only been used in the instant aka. UTC-normalized semantics so far.

From this false premise it followed that TIMESTAMPs with local semantics were a new type and did not need to be annotated with the old types to maintain compatibility. In fact, annotating them with the old types were considered to be harmful, since it would have mislead older readers into thinking that they can read TIMESTAMPs with local semantics, when in reality they would have misinterpreted them as TIMESTAMPs with instant semantics. This would have lead to a difference of several hours, corresponding to the time zone offset.

In reality, however, this misinterpretation of timestamps has already been going on for a while, since Arrow annotates local timestamps with TIMESTAMP_MILLIS or TIMESTMAP_MICROS.

To maintain forward compatibilty of local timestamps, Arrow annotates them with the legacy timestamp logical types. However, the Java library considers these logical types to be incompatible and discards the new type in favour of the legacy ones (since doing the other way around would change the behaviour). Parquet-mr should be updated so that it accepts this combination of new and old logical types.

**Reporter**: [Zoltan Ivanfi](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=zi) / @zivanfi
**Assignee**: [Nándor Kollár](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=nkollar) / @nandorKollar
#### PRs and other links:
- [Discussion on the mailing list](https://lists.apache.org/thread.html/04b9a8a156939adae1aa4cf2d4e3abd2137f541cbf92ade6d775dc06@%3Cdev.parquet.apache.org%3E)

**Note**: *This issue was originally created as [PARQUET-1628](https://issues.apache.org/jira/browse/PARQUET-1628). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Lies zuerst das Issue und die verknüpfte Mailinglisten-Diskussion, und verfolge dann, wie parquet-java logische Zeitstempeltypen behandelt und wie die Kompatibilitätsprüfungen der Java-Bibliothek funktionieren. Bestätige das Verhalten für lokale Zeitstempel von Arrow, die mit TIMESTAMP_MILLIS oder TIMESTAMP_MICROS annotiert sind; als abgeschlossen gilt die Arbeit, wenn diese Kombination akzeptiert wird, ohne den neuen Typ zu verwerfen.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
java
Bereich
data-engineering
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.