hdfsreader读失败
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
在读hdfs集群数据的时候总是提示以下错误,是怎么回事?
`java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.fs.viewfs.ViewFs.()
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134) ~[hadoop-common-2.7.2.jar:na]
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2652) ~[hadoop-common-2.7.2.jar:na]
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) ~[hadoop-common-2.7.2.jar:na]
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) ~[hadoop-common-2.7.2.jar:na]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) ~[hadoop-common-2.7.2.jar:na]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) ~[hadoop-common-2.7.2.jar:na]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170) ~[hadoop-common-2.7.2.jar:na]
at com.alibaba.datax.plugin.reader.hdfsreader.DFSUtil.getHDFSAllFiles(DFSUtil.java:130) ~[hdfsreader-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.reader.hdfsreader.DFSUtil.getAllFiles(DFSUtil.java:119) ~[hdfsreader-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.prepare(HdfsReader.java:169) ~[hdfsreader-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.JobContainer.prepareJobReader(JobContainer.java:715) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.JobContainer.prepare(JobContainer.java:308) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:115) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.start(Engine.java:92) [datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.entry(Engine.java:171) [datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.main(Engine.java:204) [datax-core-0.0.1-SNAPSHOT.jar:na]
`
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at com.alibaba.datax.plugin.reader.hdfsreader.DFSUtil.java:130 and HdfsReader.java:169, then reproduce the failure against an HDFS cluster using the reported stack trace. Determine what causes the ViewFs constructor lookup to fail and verify completion by successfully reading the cluster data without this exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100