apache / apache/seatunnel

[Bug] Iceberg Source OR Doris Sink time zone issue

Open
#10,795 1 comment 0 reactions 1 assignee Claimed by @zhangshenghang View on GitHub
bug
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 13h
Merged PRs (30d)
203

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.

### What happened

Hi, when I try to use Spark to sync data from Iceberg to Doris, I found a timezone issue with `TIMESTAMP` fields.

Spark timezone:

```
spark-sql> SET spark.sql.session.timeZone;
spark.sql.session.timeZone Asia/Shanghai
```

Doris timezone:

```
SHOW VARIABLES LIKE 'time_zone';
| Variable_name | Value | Default_Value | Changed |
| time_zone | Asia/Shanghai | Asia/Shanghai | 0 |
```

However, the actual data written into Doris is in UTC time (i.e. offset by -8 hours).

I also tried setting the timezone in the environment config, but it does not work, for example:

```
env {
parallelism = 1
job.mode = "BATCH"
spark.sql.session.timeZone = "Etc/UTC"
}
```

But it still does not take effect.

How can I fix this issue?

### SeaTunnel Version

2.3.13

### SeaTunnel Config

```conf
env {
parallelism = 1
job.mode = "BATCH"
}
source {
Iceberg {
catalog_name = "${catalog}"
iceberg.catalog.config={
type = "rest"
warehouse = "${warehouse}"
uri = "${rest.uri}"
header.X-Iceberg-Access-Delegation = "vended-credentials"
}
namespace = "${source.iceberg.database}"
table = "xxx"
query = "SELECT xxx FROM xxx WHERE dt = '${etldate}'"
plugin_output = "iceberg"
}
}
sink {
Doris {
fenodes = "${sink.doris.fenodes}"
username = "${sink.doris.username}"
password = "${sink.doris.password}"
database = "${sink.doris.database}"
table = "xxx"
sink.label-prefix = "xxx"
data_save_mode = "CUSTOM_PROCESSING"
custom_sql = "xxx"
doris.config {
format = "json"
read_json_by_line = "true"
}
}
}
```

### Running Command

```shell
${SPARK_HOME}/bin/spark-submit --class "org.apache.seatunnel.core.starter.spark.SeaTunnelSpark" --name "SeaTunnel" --master "local" --deploy-mode "client" --jars "/opt/soft/seatunnel-2.3.13/lib/seatunnel-hadoop3-3.1.4-uber.jar,/opt/soft/seatunnel-2.3.13/lib/seatunnel-hadoop-aws.jar,/opt/soft/seatunnel-2.3.13/lib/seatunnel-transforms-v2.jar,/opt/soft/seatunnel-2.3.13/lib/mysql-connector-j-8.2.0.jar,/opt/soft/seatunnel-2.3.13/connectors/connector-iceberg-2.3.13.jar,/opt/soft/seatunnel-2.3.13/connectors/connector-doris-2.3.13.jar"
```

### Error Exception

```log
nothing happend
```

### Zeta or Flink or Spark Version

spark-3.3.4

### Java or Scala Version

jdk-11

### Screenshots

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.