apache / apache/seatunnel

[Bug] [Connector-V2-File/HDFS] NoSuchMethodError for FsTracer.get with Hadoop 3.1.1 and Kerberos

Open
#10,172 2 comments 0 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 13h
Merged PRs (30d)
203

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.

### What happened

### Problem Description
When using SeaTunnel to write data from PostgreSQL to HDFS with Kerberos authentication enabled, the task fails to start and throws the exception `java.lang.NoSuchMethodError: org.apache.hadoop.fs.FsTracer.get`. The Kerberos authentication itself is successful, but a Hadoop API incompatibility error occurs when initializing the HDFS client.

### Environment Information
* **SeaTunnel Version**: 2.3.8
* **Running Mode**: SeaTunnel Engine (Local)
* **Java Version**: 1.8.0_291
* **Source Database**: PostgreSQL
* **Target HDFS Version**: Hadoop 3.1.1 (with Kerberos enabled)
* **Client Hadoop Environment**: Not explicitly specified, suspected to have a version conflict with the built-in SeaTunnel version or the version from environment variables.

### Task Configuration File (`config/pg-to-hdfs.conf`)
```hocon
env {
job.mode = "BATCH"
execution.parallelism = 1
}

source {
Jdbc {
url = "jdbc:postgresql://157.3.xx.xx:5432/sqjw?gssEncMode=disable&sslmode=disable"
driver = "org.postgresql.Driver"
user = "root"
password = "******"
query = "SELECT * from tb limit 100"
fetch_size = 1000
result_table_name = "tb_data"
}
}

sink {
HdfsFile {
path = "/db/hive/warehouse/YBSS/test-seatunnel"
file_format_type = "orc"
fs.defaultFS = "hdfs://hacluster"
dfs.client.use.datanode.hostname = "true"
hadoop.security.authentication = "kerberos"
kerberos_principal = "Botest@HADOOP.COM"
krb5_path = "/opt/krb5.conf"
kerberos_keytab_path = "/opt/user.keytab"
# I also tried to configure using the hadoopConfig {} block, but the problem remains the same.
}
}

### SeaTunnel Version

2.3.8

### SeaTunnel Config

```conf
env {
job.mode = "BATCH"
execution.parallelism = 1
}

source {
Jdbc {
url = "jdbc:postgresql://157.3.xx.xx:5432/sqjw?gssEncMode=disable&sslmode=disable"
driver = "org.postgresql.Driver"
user = "root"
password = "******"
query = "SELECT * from tb limit 100"
fetch_size = 1000
result_table_name = "tb_data"
}
}

sink {
HdfsFile {
path = "/db/hive/warehouse/YBSS/test-seatunnel"
file_format_type = "orc"
fs.defaultFS = "hdfs://hacluster"
dfs.client.use.datanode.hostname = "true"
hadoop.security.authentication = "kerberos"
kerberos_principal = "Botest@HADOOP.COM"
krb5_path = "/opt/krb5.conf"
kerberos_keytab_path = "/opt/user.keytab"
# I also tried to configure using the hadoopConfig {} block, but the problem remains the same.
}
}
```

### Running Command

```shell
bin/seatunnel.sh -c config/pg-to-hdfs.conf -m local
```

### Error Exception

```log
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:213)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: java.lang.NoSuchMethodError: org.apache.hadoop.fs.FsTracer.get(Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/tracing/Tracer;
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:323)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:308)
at org.apache.hadoop.hdfs.DistributedFileSystem.initDFSClient(DistributedFileSystem.java:204)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:189)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.lambda$initializeWithKerberosLogin$0(HadoopFileSystemProxy.java:256)
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopLoginFactory.lambda$loginWithKerberos$0(HadoopLoginFactory.java:52)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopLoginFactory.loginWithKerberos(HadoopLoginFactory.java:50)
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.initializeWithKerberosLogin(HadoopFileSystemProxy.java:249)
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.initialize(HadoopFileSystemProxy.java:215)
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.getFileSystem(HadoopFileSystemProxy.java:171)
at org.apache.seatunnel.connectors.seatunnel.file.sink.writer.OrcWriteStrategy.getOrCreateWriter(OrcWriteStrategy.java:124)
at org.apache.seatunnel.connectors.seatunnel.file.sink.writer.OrcWriteStrategy.write(OrcWriteStrategy.java:74)
at org.apache.seatunnel.connectors.seatunnel.file.sink.BaseFileSinkWriter.write(BaseFileSinkWriter.java:134)
at org.apache.seatunnel.connectors.seatunnel.file.sink.BaseFileSinkWriter.write(BaseFileSinkWriter.java:46)
at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:249)
at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:66)
at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:39)
at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:27)
at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.handleRecord(IntermediateBlockingQueue.java:75)
at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.collect(IntermediateBlockingQueue.java:50)
at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.collect(IntermediateQueueFlowLifeCycle.java:51)
at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.collect(TransformSeaTunnelTask.java:73)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168)
at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.call(TransformSeaTunnelTask.java:78)
at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:693)
at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1018)
at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:39)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:205)
```

### Zeta or Flink or Spark Version

zeta

### Java or Scala Version

1.8.0_291

### Screenshots

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the PostgreSQL-to-HDFS configuration with Hadoop 3.1.1 and Kerberos, then inspect HadoopFileSystemProxy.java around initializeWithKerberosLogin and the Hadoop dependency versions used at runtime. Compare the FsTracer API expected by DFSClient with the available Hadoop classes. Done means the local SeaTunnel job starts and writes the configured ORC output without NoSuchMethodError.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, postgresql
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.