apache / apache/beam

[Bug]: Spark "uber jar" job server fails on cluster with NoSuchFileException

Open
#23,439 2 comments 0 reactions 0 assignees View on GitHub
bug portability spark
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

Attempting to use the Spark "uber jar" job server when not in local mode fails with a NoSuchFileException.
```
python -m apache_beam.examples.wordcount \
--output ./data_test/ \
--runner=SparkRunner \
--spark_submit_uber_jar \
--spark_master_url=spark://spark-master:7077 \
--environment_type=LOOPBACK
```

This was reported in https://stackoverflow.com/a/66342031/4278032, I was able to reproduce the issue locally.

```
ERROR:root:Exception from the cluster: java.nio.file.NoSuchFileException: /tmp/tmp029vgh_0.jar
```

A work-around is to mount some shared storage for `/tmp` on all workers and the job itself. That suggests the jar isn't copied to all places where required.

### Issue Priority

Priority: 2

### Issue Component

Component: runner-spark

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.