Fix Flink 2.x Flink SQL deployment on Yarn and Kubernetes Application mode
- Dominant language
- Java
- Stars
- 4.3k
- Forks
- 1.1k
- Avg merge
- 13h 35m
- Merged PRs (30d)
- 2
Description
## Problem
Flink SQL jobs fail to start reliably on **Yarn Application** and **Kubernetes Application** mode when using Flink 2.x (e.g. 2.0.2, 2.2.1). Symptoms include:
- Yarn Application: missing dependencies (snakeyaml), incorrect user jar / classpath, Flink 2.x `ParameterTool` API relocation, and shims JAR ordering conflicts in the fat jar.
- Kubernetes Application: SqlClient logback initialization fails under parent-first classloading; Flink SQL docker image build misses snakeyaml.
- Kubernetes Session: cluster health check did not treat session clusters like remote clusters for REST reachability.
## Expected behavior
Users should be able to create, release, start, and cancel Flink SQL applications on Yarn/K8s Application mode with Flink 2.x the same way as Flink 1.x.
## Proposed fix
- Build and submit a per-application fat jar for Yarn Application SQL jobs; include snakeyaml and correct shim merge order.
- Add `FlinkParameterToolBridge` so SqlClient works with Flink 1.x and 2.x `ParameterTool` packages.
- Load logback configuration from classpath URL streams and rewrite shaded class names (fix K8s Application SqlClient startup).
- Bundle snakeyaml into K8s Application docker images for Flink SQL.
- Stop adding shims artifacts to Yarn Application `provided.lib.dirs`.
- Treat Kubernetes Session clusters as HTTP-remotable in `FlinkClusterWatcher`.
- Increase Kubernetes session cluster deploy/shutdown wait timeout for slow environments.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the Flink SQL deployment paths for Yarn Application and Kubernetes Application mode, including SqlClient startup and FlinkClusterWatcher session handling. Review the proposed FlinkParameterToolBridge, dependency and shim packaging, logback loading, and Docker image changes. Done means Flink 2.x SQL applications can be created, released, started, and cancelled reliably in the listed modes, with session health checks and timeouts working.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, java, kubernetes
- Domain
- cloud, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100