Is it necessary to use NFS in jobManager and taskManager as both pods tries to mount same volumes?

Nobody has claimed this yet.

Hi,

We recently faced following issue:
We have a working implementation of FlinkApplication, which successfully works when we do not use any volumes and the pods fallback to host volumes. It happens for both the pods (jobManager and taskManager). Since, we follow a convention that if pods require any storage they should use volumes so do not respect pods consuming host storage (very low i.e. 20GB). So, eventually node storage fills up and task manager pod fails and get restarted again and again.

To tackle the above problem, we decided to use persistent volumes, but since `FlinkApplicationsSpec` takes `volumes` and `volumeMounts` as global parameters we couldn't manage to work with persistentVolumes as well, cause EBS doesn't support ReadWriteMany access mode. Cause both the pods tries to mount same volumes.

Now, we tried adding AWS EFS and the setup initially looked fine but we started seeing weird issue s in jobManager and it keeps failing with following error:
```
:42331/user/taskmanager_0) at ResourceManager
2020-02-19 12:05:04,076 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler - Exception occurred in REST handler: Job 1fb85a4c71afb8f57b5064221ae2b340 not found
2020-02-19 12:05:34,065 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler - Exception occurred in REST handler: Job 1fb85a4c71afb8f57b5064221ae2b340 not found
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0x7) at pc=0x00007feb9438f60e, pid=1, tid=0x00007feb903c0700
#
# JRE version: OpenJDK Runtime Environment (8.0_242-b08) (build 1.8.0_242-b08)
# Java VM: OpenJDK 64-Bit Server VM (25.242-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V [libjvm.so+0x8f960e] PerfLongVariant::sample()+0x1e
#
# Core dump written. Default location: /opt/flink/core or core.1
#
# An error report file with more information is saved as:
# /opt/flink/hs_err_pid1.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
```

Task manager logs:
```
2020-02-19 12:45:59,365 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-poc-82c345f8:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@flink-poc-82c345f8:6123]] Caused by: [java.net.ConnectException: Connection refused: flink-poc-82c345f8/10.100.36.89:6123]
2020-02-19 12:45:59,366 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@flink-poc-82c345f8:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@flink-poc-82c345f8:6123/user/resourcemanager..
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0x7) at pc=0x00007f3d1022160e, pid=1, tid=0x00007f3cb0889700
#
# JRE version: OpenJDK Runtime Environment (8.0_242-b08) (build 1.8.0_242-b08)
# Java VM: OpenJDK 64-Bit Server VM (25.242-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V [libjvm.so+0x8f960e] PerfLongVariant::sample()+0x1e
#
# Core dump written. Default location: /opt/flink/core or core.1
#
# An error report file with more information is saved as:
# /opt/flink/hs_err_pid1.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
```
Operator logs:
```
{"json":{"app_name":"flink-poc","ns":"flink-operator","phase":""},"level":"error","msg":"K8s object creation failed ingresses.extensions \"flink-poc\" already exists","ts":"2020-02-19T11:43:44Z"}
{"json":{"app_name":"flink-poc","ns":"flink-operator","phase":""},"level":"info","msg":"Jobmanager ingress already exists","ts":"2020-02-19T11:43:44Z"}
{"json":{"app_name":"flink-poc","ns":"flink-operator","phase":""},"level":"info","msg":"Logged Normal event: CreatingCluster: Creating Flink cluster for deploy d8f9e43b","ts":"2020-02-19T11:43:44Z"}
{"json":{"app_name":"flink-poc","ns":"flink-operator","phase":"ClusterStarting"},"level":"info","msg":"Handling state for application","ts":"2020-02-19T11:43:44Z"}
{"json":{"app_name":"flink-poc","ns":"flink-operator","phase":"ClusterStarting"},"level":"info","msg":"Handling state for application","ts":"2020-02-19T11:44:04Z"}
{"json":{"app_name":"flink-poc","ns":"flink-operator","phase":"ClusterStarting"},"level":"info","msg":"Handling state for application","ts":"2020-02-19T11:44:34Z"}
{"json":{"app_name":"flink-poc","ns":"flink-operator","phase":"ClusterStarting"},"level":"info","msg":"Flink cluster has started successfully","ts":"2020-02-19T11:44:34Z"}
{"json":{"app_name":"flink-poc","ns":"flink-operator","phase":"Savepointing"},"level":"info","msg":"Handling state for application","ts":"2020-02-19T11:44:34Z"}
{"json":{"app_name":"flink-poc","ns":"flink-operator","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-19T11:44:34Z"}
{"json":{"app_name":"flink-poc","ns":"flink-operator","phase":"SubmittingJob"},"level":"info","msg":"No job found for the application","ts":"2020-02-19T11:44:34Z"}
{"json":{"app_name":"flink-poc","ns":"flink-operator","phase":"SubmittingJob"},"level":"info","msg":"Logged Normal event: JobSubmitted: Flink job submitted to cluster with id 5bde3064da4a7f653e3ed539db31ed1b","ts":"2020-02-19T11:44:36Z"}
{"json":{"app_name":"flink-poc","ns":"flink-operator","phase":"SubmittingJob"},"level":"warning","msg":"Conflict while updating status","ts":"2020-02-19T11:44:36Z"}
{"json":{"app_name":"flink-poc","ns":"flink-operator","phase":"SubmittingJob"},"level":"warning","msg":"Failed to reconcile resource flink-operator/flink-poc: Operation cannot be fulfilled on flinkapplications.flink.k8s.io \"flink-poc\": the object has been modified; please apply your changes to the latest version and try again","ts":"2020-02-19T11:44:36Z"}
{"json":{"app_name":"flink-poc","ns":"flink-operator","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-19T11:44:36Z"}
{"json":{"app_name":"flink-poc","ns":"flink-operator","phase":"SubmittingJob"},"level":"warning","msg":"Found already-submitted job for application with id 5bde3064da4a7f653e3ed539db31ed1b","ts":"2020-02-19T11:44:36Z"}
{"json":{"app_name":"flink-poc","ns":"flink-operator","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-19T11:44:36Z"}
{"json":{"app_name":"flink-poc","ns":"flink-operator","phase":"SubmittingJob"},"level":"info","msg":"Handling state for application","ts":"2020-02-19T11:45:04Z"}
{"json":{"app_name":"flink-poc","ns":"flink-operator","phase":"SubmittingJob"},"level":"info","msg":"RESTY 2020/02/19 11:45:05 ERROR Get http://flink-poc-d8f9e43b.flink-operator:8081/overview: dial tcp 10.100.5.87:8081: connect: connection refused, Attempt 1","ts":"2020-02-19T11:45:05Z"}
```

The logs of operator is completely making sense that jobManager is not running and the operator is not able to connect to it. But, the question why we can't specify separate volumes for jobManager and taskManager, is it necessary?

Googling `SIGBUS` with flink also didn't help. Either we are doing something wrong with EFS or we should look somewhere else. This is a working application which completely runs fine when the pods doesn't have any volume and if the host has enough storage.

Let me know, if you guys need more info, but we are stuck here and would love to get it fixed.

Start with the FlinkApplicationsSpec volume and volumeMounts behavior and reproduce the issue using separate jobManager and taskManager storage. Compare the operator, jobManager, and taskManager logs to determine whether shared volume configuration or EFS is causing the SIGBUS and connection failures; done means identifying the cause and documenting supported storage configuration.

Written by the indexing model from the issue text.