lyft / lyft/flinkk8soperator

Beam batch job without sink stuck in SubmittingJob state

Open
#189 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
581
Forks
153
PR merge metrics
No merged PRs in 30d

Description

Hi,

When creating a FlinkApplication running a beam batch job which does not have a sink, the application is stucked in the `SubmittingJob` state.

This is due to the fact that when translating a beam batch job, flink is terminating all dangling datasets with the `DiscardingOutputFormat` sink:
https://github.com/apache/beam/blob/bbc0c18c8b86794d1632e25128b5d80184112bd7/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPipelineTranslator.java#L52-L54

It seems that this dummy sink remains in a `CREATED` state until all elements have been processed before switching to a `FINISHED` state.

As the Flink controller is checking that all tasks are running before switching the FlinkApplication state from `SubmittingJob` to `Running`, it never happens in this case:
https://github.com/lyft/flinkk8soperator/blob/4c377a32b3fbfa8c7b91fd72a3a01ccfb3f669a5/pkg/controller/flinkapplication/flink_state_machine.go#L588-L593

Should we relax this condition ? If at least one task is running and the Flink job status is running then we can consider the FlinkApplication as running. What do you think ?

Thanks !

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in pkg/controller/flinkapplication/flink_state_machine.go around lines 588-593 and trace the SubmittingJob-to-Running transition. Reproduce or inspect the Beam batch job with no sink, then determine and verify the expected transition when the Flink job is running even if a dangling task remains in CREATED.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.