Exception occurred in REST handler: Job X not found
- Dominant language
- Go
- Stars
- 581
- Forks
- 153
- PR merge metrics
- No merged PRs in 30d
Description
Hey,
I am using the operator in version `docker.io/lyft/flinkk8soperator:1355d206b5fb4efd6f6e4ccf24085a87a29443c5`.
Running ok aws eks version 1.21.
Sometimes The job manager floods the log with this message and when it starts, I am unable to redeploy the flinkapp without reaching the "DeployFailed" state
log: `2022-07-04 06:03:35,466 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [] - Exception occurred in REST handler: Job not found`
at the same time, task manager does not have any logs in it (makes sense)
in the operator logs I see the below log for multiple flink apps:
```{"json":{"app_name":"esp-process-666","ns":"int-streaming","phase":"Running"},"level":"warning","msg":"Failed to reconcile resource /: GetJobOverview call failed with status 404 Not Found and message ''","ts":"2022-07-04T06:08:35Z"}```
is this a known issue?
how do I recover from this without deleting and redeploying the flink app?
Contributor guide
No contributing guide indexed for this repository
Research direction
No source file or test is named. Start with the operator reconciliation path that calls GetJobOverview and compare it with the Flink REST handler's 404 log; done means identifying why the application reaches DeployFailed and documenting or implementing recovery without deleting and redeploying it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, go, kubernetes
- Domain
- backend, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100