alibaba / alibaba/DataX

hdfsreader读失败

Open
#483 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

在读hdfs集群数据的时候总是提示以下错误,是怎么回事?
`java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.fs.viewfs.ViewFs.()
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134) ~[hadoop-common-2.7.2.jar:na]
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2652) ~[hadoop-common-2.7.2.jar:na]
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) ~[hadoop-common-2.7.2.jar:na]
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) ~[hadoop-common-2.7.2.jar:na]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) ~[hadoop-common-2.7.2.jar:na]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) ~[hadoop-common-2.7.2.jar:na]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170) ~[hadoop-common-2.7.2.jar:na]
at com.alibaba.datax.plugin.reader.hdfsreader.DFSUtil.getHDFSAllFiles(DFSUtil.java:130) ~[hdfsreader-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.reader.hdfsreader.DFSUtil.getAllFiles(DFSUtil.java:119) ~[hdfsreader-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.prepare(HdfsReader.java:169) ~[hdfsreader-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.JobContainer.prepareJobReader(JobContainer.java:715) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.JobContainer.prepare(JobContainer.java:308) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:115) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.start(Engine.java:92) [datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.entry(Engine.java:171) [datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.main(Engine.java:204) [datax-core-0.0.1-SNAPSHOT.jar:na]
`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at com.alibaba.datax.plugin.reader.hdfsreader.DFSUtil.java:130 and HdfsReader.java:169, then reproduce the failure against an HDFS cluster using the reported stack trace. Determine what causes the ViewFs constructor lookup to fail and verify completion by successfully reading the cluster data without this exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.