lyft / lyft/flinkk8soperator

Deletion and recreation of FlinkApplication with the same name tries to continue the old job

Open
#166 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
581
Forks
153
PR merge metrics
No merged PRs in 30d

Description

I observed the following behaviour during deletion and recreation of a FlinkApplication custom resource:

1. I create a FlinkApplication with name x. The Flink job cluster is correctly created and running.
2. I delete the FlinkApplication x. The Flink job cluster is correctly removed.
3. I create a new FlinkApplication with the same name x. The Flink job cluster is created correctly, but the job is not started. The job manager log contains the error message
```
Exception occurred in REST handler: Job 2cb963c30af93d419aa54d562145fe55 not found
```
Here, 2cb963c30af93d419aa54d562145fe55 is the id of the job running in the Flink job cluster created in step 1.
I would have expected that the Flink job cluster only tries to continue the old job when an update to the FlinkApplication was done.

The deleteMode of the FlinkApplication x is None.
I'm running the flink-operator in version 0.4.0.

I would like to know which FlinkApplication configuration would be required so that deleting and recreating a FlinkApplication custom resource works as expected.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing deletion and recreation of a FlinkApplication with the same name using deleteMode None, then inspect how the operator handles job recovery state across resource deletion. Compare the new resource's startup request with the old job ID shown in the job manager error. Done means the recreated resource starts its new job instead of trying to continue the deleted job.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.