lyft / lyft/flinkk8soperator

Exception occurred in REST handler: Job X not found

Open
#256 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
581
Forks
153
PR merge metrics
No merged PRs in 30d

Description

Hey,

I am using the operator in version `docker.io/lyft/flinkk8soperator:1355d206b5fb4efd6f6e4ccf24085a87a29443c5`.
Running ok aws eks version 1.21.

Sometimes The job manager floods the log with this message and when it starts, I am unable to redeploy the flinkapp without reaching the "DeployFailed" state

log: `2022-07-04 06:03:35,466 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler [] - Exception occurred in REST handler: Job not found`

at the same time, task manager does not have any logs in it (makes sense)

in the operator logs I see the below log for multiple flink apps:
```{"json":{"app_name":"esp-process-666","ns":"int-streaming","phase":"Running"},"level":"warning","msg":"Failed to reconcile resource /: GetJobOverview call failed with status 404 Not Found and message ''","ts":"2022-07-04T06:08:35Z"}```

is this a known issue?
how do I recover from this without deleting and redeploying the flink app?

Contributor guide

No contributing guide indexed for this repository

Research direction

No source file or test is named. Start with the operator reconciliation path that calls GetJobOverview and compare it with the Flink REST handler's 404 log; done means identifying why the application reaches DeployFailed and documenting or implementing recovery without deleting and redeploying it.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, go, kubernetes
Domain
backend, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.