apache / apache/kyuubi

[Bug] Error while compiling statement: FAILED: ParseException line 1:37 cannot recognize input near '.' 'id1' 'BIGINT' in column type

Open
#4,051 14 comments 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

I use the hive jdbc dialect plugin like this:https://kyuubi.readthedocs.io/en/latest/extensions/engines/spark/jdbc-dialect.html
But I'm running into some issues in my spark program.

I build kyuubi-extension-spark-jdbc-dialect_2.12-1.7.0-SNAPSHOT.jar from branch 'master', put it into $SPARK_HOEM/jars.

show my code:
```java

import org.apache.spark.SparkConf;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SaveMode;
import org.apache.spark.sql.SparkSession;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class HiveDemo {

private static final Logger log = LoggerFactory.getLogger(HiveDemo.class);

public static void main(String[] args) {
SparkConf sparkConf = new SparkConf();
sparkConf.set("spark.sql.extensions", "org.apache.spark.sql.dialect.KyuubiSparkJdbcDialectExtension");
SparkSession sparkSession = SparkSession.builder()
.config(sparkConf)
.enableHiveSupport()
.getOrCreate();
boolean flag = true;
try {
Dataset rowDataset = sparkSession.read()
.format("jdbc")
.option("url", "jdbc:hive2://10.58.11.24:10000/ibond")
.option("dbtable", "a1000w_30d")
.option("user", "tdops")
.option("password", "None")
.option("driver", "org.apache.hive.jdbc.HiveDriver")
.load();
rowDataset.show();
rowDataset.limit(20)
.write()
.format("jdbc")
.mode(SaveMode.Overwrite)
.option("driver", "org.apache.hive.jdbc.HiveDriver")
.option("url", "jdbc:hive2://10.58.11.24:10000/ibond")
.option("dbtable", "mgy_test_1")
.option("user", "tdops")
.option("password", "None")
.save();
log.info("Hive Demo Success.");
} catch (Exception ex) {
ex.printStackTrace();
flag = false;
throw new RuntimeException("Hive Demo Failed.", ex);
} finally {
sparkSession.stop();
System.exit(flag ? 0 : -1);
}
}
}

```

console output:
```
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:37 cannot recognize input near '.' 'id1' 'BIGINT' in column type

at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)

at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:253)

at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:313)

at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:253)

at org.apache.hive.jdbc.HiveStatement.executeUpdate(HiveStatement.java:490)

at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.executeStatement(JdbcUtils.scala:993)

at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.createTable(JdbcUtils.scala:878)

at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:81)

at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)

at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)

at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)

at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90)

at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180)

at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)

at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)

at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:132)

at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:131)

at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989)

at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)

at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)

at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)

at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)

at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)

at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989)

at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:438)

at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:415)

at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:301)

at cn.tongdun.sparkdatahandler.HiveDemo.main(HiveDemo.java:46)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.base/java.lang.reflect.Method.invoke(Unknown Source)

at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)

at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)

at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)

at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)

at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)

at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:37 cannot recognize input near '.' 'id1' 'BIGINT' in column type

at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)

at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)

at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)

at org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)

at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)

at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)

at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)

at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:530)

at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)

at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)

at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)

at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)

at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)

at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.parse.ParseException:line 1:37 cannot recognize input near '.' 'id1' 'BIGINT' in column type

at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:211)

at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:77)

at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:70)

at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468)

at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)

at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1295)

at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:204)

... 15 more
```

### Affects Version(s)

master

### Kyuubi Server Log Output

_No response_

### Kyuubi Engine Log Output

_No response_

### Kyuubi Server Configurations

_No response_

### Kyuubi Engine Configurations

_No response_

### Additional context

_No response_

### Are you willing to submit PR?

- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

Start at KyuubiSparkJdbcDialectExtension, using the failing Spark JdbcUtils.createTable path and the Hive JDBC error as the reproduction entry point. Inspect the generated CREATE TABLE statement for the invalid column type syntax, then verify that writing with SaveMode.Overwrite succeeds against Hive without the ParseException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.