apache / apache/beam

can't read/write hdfs in Flink CLUSTER(Standalone)

Open
#18,553 0 comments 0 reactions 0 assignees View on GitHub
bug flink P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 5h
Merged PRs (30d)
204

Description

i just write a simple demo like:

```

Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://localhost:9000");
//other
codes
p.apply("ReadLines", TextIO.read().from("hdfs://localhost:9000/tmp/words"))

.apply(TextIO.write().to("hdfs://localhost:9000/tmp/hdfsout"));

```

it works in flink local model with cmd:

```

mvn exec:java -Dexec.mainClass=com.joe.FlinkWithHDFS -Pflink-runner -Dexec.args="--runner=FlinkRunner
--filesToStage=target/flinkBeam-2.2.0-SNAPSHOT-shaded.jar"

```

but not works in CLUSTER mode:

```

mvn exec:java -Dexec.mainClass=com.joe.FlinkWithHDFS -Pflink-runner -Dexec.args="--runner=FlinkRunner
--filesToStage=target/flinkBeam-2.2.0-SNAPSHOT-shaded.jar --flinkMaster=localhost:6123 "

```

it seems the flink cluster regard the hdfs as local file system.
The input log from flink-jobmanger.log is:

```

2017-09-27 20:17:37,962 INFO org.apache.flink.runtime.jobmanager.JobManager - Successfully
ran initialization on master in 136 ms.
2017-09-27 20:17:37,968 INFO org.apache.beam.sdk.io.FileBasedSource
- {color:red}Filepattern hdfs://localhost:9000/tmp/words2 matched 0 files with
total size 0{color}
2017-09-27 20:17:37,968 INFO org.apache.beam.sdk.io.FileBasedSource
- Splitting filepattern hdfs://localhost:9000/tmp/words2 into bundles of size 0 took 0 ms
and produced 0 files a
nd 0 bundles

```

The output error message is :

```

Caused by: java.lang.ClassCastException: {color:red}org.apache.beam.sdk.io.hdfs.HadoopResourceId cannot
be cast to org.apache.beam.sdk.io.LocalResourceId{color}
at org.apache.beam.sdk.io.LocalFileSystem.create(LocalFileSystem.java:77)

at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:256)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:243)

at org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:922)
at org.apache.beam.sdk.io.FileBasedSink$Writer.openUnwindowed(FileBasedSink.java:884)

at org.apache.beam.sdk.io.WriteFiles.finalizeForDestinationFillEmptyShards(WriteFiles.java:909)

at org.apache.beam.sdk.io.WriteFiles.access$900(WriteFiles.java:110)
at org.apache.beam.sdk.io.WriteFiles$2.processElement(WriteFiles.java:858)

```

can somebody help me, i've try all the way just can't work it out [cry]
https://issues.apache.org/jira/browse/BEAM-2457

Imported from Jira [BEAM-2995](https://issues.apache.org/jira/browse/BEAM-2995). Original Jira may contain additional context.
Reported by: huangjianhuang.

Contributor guide

Open the contributing guide

Research direction

Start with the Beam FileSystems, HadoopResourceId, LocalResourceId, and FileBasedSink stack frames, then compare behavior between the local and Flink cluster commands shown. Done means the cluster can read from and write to the hdfs:// paths without treating them as local files or raising the reported ClassCastException.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.