apache / apache/seatunnel

[Bug] [seatunnel zeta] LocalFile synchronization Hive failed

Open
#5,771 4 comments 0 reactions 0 assignees View on GitHub
bug Waiting for users feedback
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.

### What happened

The local file(_a.txt_) exists on the node of seatunnel-cluster, with the content being
1000#2000
3000#4000

The error message is as follows:
![image](https://github.com/apache/seatunnel/assets/8542648/5d8a3bd7-d470-43c4-96d4-2092992ef95e)

### SeaTunnel Version

2.3.1

### SeaTunnel Config

```conf
env {
job.mode = "BATCH"
execution.parallelism = 1
job.name="judata-etl"
}
source {
LocalFile {
result_table_name = "temp_source_table"
schema {
fields {
name=string
age=string
}
}
path = "/usr/linkis/tmp/a.txt"
file_format_type = "text"
#fs.defaultFS="hdfs://hive-1.cdh.test:8020"
delimiter="#"
read_columns = [name, age]
}
}
transform {
FieldMapper {
source_table_name ="temp_source_table"
result_table_name ="temp_sink_table"
field_mapper = {
name=productcode
age=regioncode
}
}
}
sink {

Hive {
table_name="default.sxtnopart"
metastore_uri="thrift://xxxxx:xxx"
source_table_name = "temp_sink_table"
}
}
```

### Running Command

```shell
/opt/apache-seatunnel-incubating-2.3.1/bin/seatunnel.sh -c file_config -cn seatunnel_cluster
```

### Error Exception

```log
java.io.InvalidClassException: org.apache.seatunnel.connectors.seatunnel.file.sink.writer.OrcWriteStrategy; local class incompatible: stream classdesc serialVersionUID = 2763560317396106200, local class serialVersionUID = 2573392164479111009
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:687)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1880)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1746)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2037)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1568)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2282)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2206)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2064)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1568)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2282)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2206)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2064)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1568)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:428)
at com.hazelcast.internal.serialization.impl.defaultserializers.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:92)
at com.hazelcast.internal.serialization.impl.defaultserializers.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:85)
at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:44)
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:349)
```

### Zeta or Flink or Spark Version

zeta

### Java or Scala Version

1.8

### Screenshots

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the OrcWriteStrategy serialization failure in the reported stack trace and compare the serialized and local serialVersionUID values across the SeaTunnel cluster. Reproduce the LocalFile-to-Hive job using the provided configuration and Java 8 environment; done means the job completes without the InvalidClassException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.