apache / apache/kyuubi

[TASK][EASY] Support using flink-table-planner explicitly

Open
#6,229 6 comments 0 reactions 1 assignee Claimed by @link3280 View on GitHub
kind:feature priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the feature

[#Table Planner and Table Planner Loader ](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/configuration/advanced/#table-planner-and-table-planner-loader)

Starting from Flink 1.15, the distribution contains two planners:

- flink-table-planner_2.12-1.20-SNAPSHOT.jar, in /opt, contains the query planner
- flink-table-planner-loader-1.20-SNAPSHOT.jar, loaded by default in /lib, contains the query planner hidden behind an isolated classpath (you won’t be able to address any io.apache.flink.table.planner directly)

**The two planners cannot co-exist at the same time in the classpath. If you load both of them in /lib your Table Jobs will fail.**

Because Dinky and hudi use flink-table-planner, so I have to put flink-table-planner into flink /lib, however Apache kyuubi will check flink-table-planner-loader in path. So could you adapt to use flink-table-planner_2.12?

If use flink-table-planner, error will be
```
Error: Error operating ExecuteStatement: org.apache.flink.table.api.TableException: Error obtaining Flink planner module ClassLoader. Make sure a flink-table-planner-loader.jar is on the classpath
at org.apache.kyuubi.engine.flink.operation.ExecuteStatement.getPlannerModuleClassLoader(ExecuteStatement.scala:117)
at org.apache.kyuubi.engine.flink.operation.ExecuteStatement.parseExtendedStatement(ExecuteStatement.scala:90)
at org.apache.kyuubi.engine.flink.operation.ExecuteStatement.executeStatement(ExecuteStatement.scala:69)
at org.apache.kyuubi.engine.flink.operation.ExecuteStatement.runInternal(ExecuteStatement.scala:62)
at org.apache.kyuubi.operation.AbstractOperation.run(AbstractOperation.scala:173)
at org.apache.kyuubi.session.AbstractSession.runOperation(AbstractSession.scala:100)
at org.apache.kyuubi.session.AbstractSession.$anonfun$executeStatement$1(AbstractSession.scala:130)
at org.apache.kyuubi.session.AbstractSession.withAcquireRelease(AbstractSession.scala:81)
at org.apache.kyuubi.session.AbstractSession.executeStatement(AbstractSession.scala:127)
at org.apache.kyuubi.service.AbstractBackendService.executeStatement(AbstractBackendService.scala:66)
at org.apache.kyuubi.service.TFrontendService.ExecuteStatement(TFrontendService.scala:253)
at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
at org.apache.kyuubi.shaded.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.kyuubi.shaded.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:35)
at org.apache.kyuubi.shaded.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.NoSuchMethodException: Cannot find method: getInstance
at org.apache.kyuubi.util.reflect.DynMethods$Builder.buildChecked(DynMethods.java:473)
at org.apache.kyuubi.util.reflect.DynMethods$Builder.buildStaticChecked(DynMethods.java:500)
at org.apache.kyuubi.engine.flink.operation.ExecuteStatement.getPlannerModuleClassLoader(ExecuteStatement.scala:105)
... 19 more (state=,code=0)
```

### Motivation

_No response_

### Describe the solution

_No response_

### Additional context

_No response_

### Are you willing to submit PR?

- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [X] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.