apache / apache/iceberg

Allow for v3 date -> time promotion

Open
#16,083 8 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

In https://iceberg.apache.org/spec/#schema-evolution point out in v3 table ,we can promotion date to timestamp and timestamp_ns. But when I try it I get below.

Flink:

```java
@TestTemplate
public void testDateToTimestamp() throws Exception {
String tableName = "test_insert_into_timestamp";
sql(
"CREATE TABLE %s(id INT, data date) WITH ('write.format.default'='%s','format-version'='3')",
tableName, format.name());

try {
Table table =
validationCatalog.loadTable(TableIdentifier.of(icebergNamespace, tableName));

sql("INSERT INTO %s SELECT 1, DATE '2024-01-01'", tableName);
sql("INSERT INTO %s SELECT 2, DATE '2024-01-02'", tableName);
sql("INSERT INTO %s SELECT 3, DATE '2024-01-03'", tableName);

table
.updateSchema()
.updateColumn("data", Types.TimestampType.withoutZone())
.commit();

table.refresh();

sql("SELECT * FROM %s ORDER BY id", tableName);
} finally {
sql("DROP TABLE IF EXISTS %s.%s", flinkDatabase, tableName);
}
}
```

```shell
Cannot change column type: data: date -> timestamp
java.lang.IllegalArgumentException: Cannot change column type: data: date -> timestamp
at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:463)
at org.apache.iceberg.SchemaUpdate.updateColumn(SchemaUpdate.java:285)
at org.apache.iceberg.flink.TestFlinkTableSink.testDateToTimestamp(TestFlinkTableSink.java:341)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/java.util.Optional.ifPresent(Optional.java:178)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
```

Spark:

```java

@TestTemplate
public void testDateToTimestamp() {
String tableName = tableName("date_to_timestamp_table");
sql(
"CREATE TABLE %s (id INT NOT NULL, dt DATE) "
+ "USING iceberg TBLPROPERTIES ('format-version'='3')",
tableName);
sql("INSERT INTO %s VALUES (1, DATE '2023-01-01')", tableName);
sql("ALTER TABLE %s ALTER COLUMN dt TYPE timestamp_ntz", tableName);
List sql = sql("select * from %s", tableName);
System.out.println(sql);
}
```

```shell

Cannot change column type: dt: date -> timestamptz
java.lang.IllegalArgumentException: Cannot change column type: dt: date -> timestamptz
at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:463)
at org.apache.iceberg.SchemaUpdate.updateColumn(SchemaUpdate.java:285)
at org.apache.iceberg.spark.Spark3Util.applySchemaChanges(Spark3Util.java:179)
at org.apache.iceberg.spark.SparkCatalog.commitChanges(SparkCatalog.java:781)
at org.apache.iceberg.spark.SparkCatalog.alterTable(SparkCatalog.java:294)
at org.apache.spark.sql.execution.datasources.v2.AlterTableExec.run(AlterTableExec.scala:40)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
```

I think we should implement this feature.

### Query engine

None

### Willingness to contribute

- [ ] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start in SchemaUpdate.java at updateColumn, then compare the Flink TestFlinkTableSink.testDateToTimestamp reproduction with Spark3Util.applySchemaChanges. Confirm the v3 date-to-timestamp and timestamp_ns promotions in both examples, and consider the work done when the schema updates and subsequent SELECT results succeed without the type-change exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.