[Bug] [Connector-V2-File/HDFS] NoSuchMethodError for FsTracer.get with Hadoop 3.1.1 and Kerberos
- Dominant language
- Java
- Stars
- 9.7k
- Forks
- 2.4k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 203
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
### Problem Description
When using SeaTunnel to write data from PostgreSQL to HDFS with Kerberos authentication enabled, the task fails to start and throws the exception `java.lang.NoSuchMethodError: org.apache.hadoop.fs.FsTracer.get`. The Kerberos authentication itself is successful, but a Hadoop API incompatibility error occurs when initializing the HDFS client.
### Environment Information
* **SeaTunnel Version**: 2.3.8
* **Running Mode**: SeaTunnel Engine (Local)
* **Java Version**: 1.8.0_291
* **Source Database**: PostgreSQL
* **Target HDFS Version**: Hadoop 3.1.1 (with Kerberos enabled)
* **Client Hadoop Environment**: Not explicitly specified, suspected to have a version conflict with the built-in SeaTunnel version or the version from environment variables.
### Task Configuration File (`config/pg-to-hdfs.conf`)
```hocon
env {
job.mode = "BATCH"
execution.parallelism = 1
}
source {
Jdbc {
url = "jdbc:postgresql://157.3.xx.xx:5432/sqjw?gssEncMode=disable&sslmode=disable"
driver = "org.postgresql.Driver"
user = "root"
password = "******"
query = "SELECT * from tb limit 100"
fetch_size = 1000
result_table_name = "tb_data"
}
}
sink {
HdfsFile {
path = "/db/hive/warehouse/YBSS/test-seatunnel"
file_format_type = "orc"
fs.defaultFS = "hdfs://hacluster"
dfs.client.use.datanode.hostname = "true"
hadoop.security.authentication = "kerberos"
kerberos_principal = "Botest@HADOOP.COM"
krb5_path = "/opt/krb5.conf"
kerberos_keytab_path = "/opt/user.keytab"
# I also tried to configure using the hadoopConfig {} block, but the problem remains the same.
}
}
### SeaTunnel Version
2.3.8
### SeaTunnel Config
```conf
env {
job.mode = "BATCH"
execution.parallelism = 1
}
source {
Jdbc {
url = "jdbc:postgresql://157.3.xx.xx:5432/sqjw?gssEncMode=disable&sslmode=disable"
driver = "org.postgresql.Driver"
user = "root"
password = "******"
query = "SELECT * from tb limit 100"
fetch_size = 1000
result_table_name = "tb_data"
}
}
sink {
HdfsFile {
path = "/db/hive/warehouse/YBSS/test-seatunnel"
file_format_type = "orc"
fs.defaultFS = "hdfs://hacluster"
dfs.client.use.datanode.hostname = "true"
hadoop.security.authentication = "kerberos"
kerberos_principal = "Botest@HADOOP.COM"
krb5_path = "/opt/krb5.conf"
kerberos_keytab_path = "/opt/user.keytab"
# I also tried to configure using the hadoopConfig {} block, but the problem remains the same.
}
}
```
### Running Command
```shell
bin/seatunnel.sh -c config/pg-to-hdfs.conf -m local
```
### Error Exception
```log
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:213)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: java.lang.NoSuchMethodError: org.apache.hadoop.fs.FsTracer.get(Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/tracing/Tracer;
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:323)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:308)
at org.apache.hadoop.hdfs.DistributedFileSystem.initDFSClient(DistributedFileSystem.java:204)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:189)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.lambda$initializeWithKerberosLogin$0(HadoopFileSystemProxy.java:256)
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopLoginFactory.lambda$loginWithKerberos$0(HadoopLoginFactory.java:52)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopLoginFactory.loginWithKerberos(HadoopLoginFactory.java:50)
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.initializeWithKerberosLogin(HadoopFileSystemProxy.java:249)
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.initialize(HadoopFileSystemProxy.java:215)
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.getFileSystem(HadoopFileSystemProxy.java:171)
at org.apache.seatunnel.connectors.seatunnel.file.sink.writer.OrcWriteStrategy.getOrCreateWriter(OrcWriteStrategy.java:124)
at org.apache.seatunnel.connectors.seatunnel.file.sink.writer.OrcWriteStrategy.write(OrcWriteStrategy.java:74)
at org.apache.seatunnel.connectors.seatunnel.file.sink.BaseFileSinkWriter.write(BaseFileSinkWriter.java:134)
at org.apache.seatunnel.connectors.seatunnel.file.sink.BaseFileSinkWriter.write(BaseFileSinkWriter.java:46)
at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:249)
at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:66)
at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:39)
at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:27)
at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.handleRecord(IntermediateBlockingQueue.java:75)
at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.collect(IntermediateBlockingQueue.java:50)
at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.collect(IntermediateQueueFlowLifeCycle.java:51)
at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.collect(TransformSeaTunnelTask.java:73)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168)
at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.call(TransformSeaTunnelTask.java:78)
at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:693)
at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1018)
at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:39)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:205)
```
### Zeta or Flink or Spark Version
zeta
### Java or Scala Version
1.8.0_291
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the PostgreSQL-to-HDFS configuration with Hadoop 3.1.1 and Kerberos, then inspect HadoopFileSystemProxy.java around initializeWithKerberosLogin and the Hadoop dependency versions used at runtime. Compare the FsTracer API expected by DFSClient with the available Hadoop classes. Done means the local SeaTunnel job starts and writes the configured ORC output without NoSuchMethodError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, postgresql
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100