dbt-labs / dbt-labs/dbt-adapters
[Regression] dbt-spark-thrift docker error
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
### Is this a regression in a recent version of dbt-spark?
- [X] I believe this is a regression in dbt-spark functionality
- [X] I have searched the existing issues, and I could not find an existing issue for this regression
### Current Behavior
- git clone the master branch.
- docker-compose up -d
- docker container launches successfully.
- try to connect using jdbc with tool such as dbeaver or dbvisualizer or run a simple dbt test - or you can exec into the spark docker container and exec "spark-sql" and at the command prompt type "select 1;" and you will see the same error.
Notice you cannot connect and the following error is in docker logs:
ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.8ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.825/01/02 18:59:45
Exception in thread "HiveServer2-Handler-Pool: Thread-82" java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(Lorg/antlr/v4/runtime/ParserRuleContext;Lscala/Function0;)Ljava/lang/Object;
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieSqlCommonAstBuilder.visitSingleStatement(HoodieSqlCommonAstBuilder.scala:37)
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieSqlCommonAstBuilder.visitSingleStatement(HoodieSqlCommonAstBuilder.scala:31)
2025-01-02 11:40:04 at org.apache.hudi.spark.sql.parser.HoodieSqlCommonParser$SingleStatementContext.accept(HoodieSqlCommonParser.java:107)
2025-01-02 11:40:04 at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18)
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieCommonSqlParser.$anonfun$parsePlan$1(HoodieCommonSqlParser.scala:42)
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieCommonSqlParser.parse(HoodieCommonSqlParser.scala:84)
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieCommonSqlParser.parsePlan(HoodieCommonSqlParser.scala:41)
2025-01-02 11:40:04 at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:620)
2025-01-02 11:40:04 at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
2025-01-02 11:40:04 at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:620)
2025-01-02 11:40:04 at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
2025-01-02 11:40:04 at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
2025-01-02 11:40:04 at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:651)
2025-01-02 11:40:04 at org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.openSession(SparkSQLSessionManager.scala:71)
2025-01-02 11:40:04 at org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:203)
2025-01-02 11:40:04 at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:370)
2025-01-02 11:40:04 at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:242)
2025-01-02 11:40:04 at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1497)
2025-01-02 11:40:04 at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1482)
2025-01-02 11:40:04 at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
2025-01-02 11:40:04 at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
2025-01-02 11:40:04 at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:52)
2025-01-02 11:40:04 at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
2025-01-02 11:40:04 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2025-01-02 11:40:04 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2025-01-02 11:40:04 at java.lang.Thread.run(Thread.java:750)
### Expected/Previous Behavior
docker-compose up -d should launch a docker container that can be connected to!
### Steps To Reproduce
- git clone the master branch.
- docker-compose up -d
- docker container launches successfully.
- try to connect using jdbc with tool such as dbeaver or dbvisualizer or run a simple dbt test - or you can exec into the spark docker container and exec "spark-sql" and at the command prompt type "select 1;" and you will see the same error.
### Relevant log output
ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.8ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.825/01/02 18:59:45
Exception in thread "HiveServer2-Handler-Pool: Thread-82" java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(Lorg/antlr/v4/runtime/ParserRuleContext;Lscala/Function0;)Ljava/lang/Object;
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieSqlCommonAstBuilder.visitSingleStatement(HoodieSqlCommonAstBuilder.scala:37)
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieSqlCommonAstBuilder.visitSingleStatement(HoodieSqlCommonAstBuilder.scala:31)
2025-01-02 11:40:04 at org.apache.hudi.spark.sql.parser.HoodieSqlCommonParser$SingleStatementContext.accept(HoodieSqlCommonParser.java:107)
2025-01-02 11:40:04 at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18)
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieCommonSqlParser.$anonfun$parsePlan$1(HoodieCommonSqlParser.scala:42)
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieCommonSqlParser.parse(HoodieCommonSqlParser.scala:84)
2025-01-02 11:40:04 at org.apache.spark.sql.parser.HoodieCommonSqlParser.parsePlan(HoodieCommonSqlParser.scala:41)
2025-01-02 11:40:04 at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:620)
2025-01-02 11:40:04 at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
2025-01-02 11:40:04 at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:620)
2025-01-02 11:40:04 at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
2025-01-02 11:40:04 at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
2025-01-02 11:40:04 at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:651)
2025-01-02 11:40:04 at org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.openSession(SparkSQLSessionManager.scala:71)
2025-01-02 11:40:04 at org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:203)
2025-01-02 11:40:04 at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:370)
2025-01-02 11:40:04 at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:242)
2025-01-02 11:40:04 at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1497)
2025-01-02 11:40:04 at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1482)
2025-01-02 11:40:04 at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
2025-01-02 11:40:04 at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
2025-01-02 11:40:04 at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:52)
2025-01-02 11:40:04 at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
2025-01-02 11:40:04 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2025-01-02 11:40:04 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2025-01-02 11:40:04 at java.lang.Thread.run(Thread.java:750)
```
### Environment
```markdown
- OS: mac os
- Python:
- dbt-core (working version): unrelated
- dbt-spark (working version): unknown, but I assume this was working at some point
- dbt-core (regression version): unrelated
- dbt-spark (regression version): master branch docker compose file
```
### Additional Context
Looking at spark 3.3.2 docker file, it looks like java 11 is required?
https://github.com/apache/spark-docker/blob/master/3.3.2/scala2.12-java11-python3-ubuntu/Dockerfile
However, when I replaced dbt-spark spark.Dockerfile with 11, I still got the same error.
To fix this I had to completely replace the spark.Dockerfile with the spark docker file (above) contents, and also add the postgres jar as in original spark.Dockerfile.
After using the "modified" https://github.com/apache/spark-docker/blob/master/3.3.2/scala2.12-java11-python3-ubuntu/Dockerfile, it worked as expected. Here's my modified docker file (I added the postgres and also set PATH)
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
FROM eclipse-temurin:11-jre-focal
ARG spark_uid=185
RUN groupadd --system --gid=${spark_uid} spark && \
useradd --system --uid=${spark_uid} --gid=spark spark
RUN set -ex && \
apt-get update && \
apt-get install -y wget netcat-openbsd procps libpostgresql-jdbc-java && \
ln -s /lib /lib64 && \
apt install -y gnupg2 wget bash tini libc6 libpam-modules krb5-user libnss3 procps net-tools gosu && \
apt install -y python3 python3-pip && \
mkdir -p /opt/spark && \
mkdir /opt/spark/python && \
mkdir -p /opt/spark/examples && \
mkdir -p /opt/spark/work-dir && \
touch /opt/spark/RELEASE && \
chown -R spark:spark /opt/spark && \
rm /bin/sh && \
ln -sv /bin/bash /bin/sh && \
echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su && \
chgrp root /etc/passwd && chmod ug+rw /etc/passwd && \
rm -rf /var/cache/apt/* && \
rm -rf /var/lib/apt/lists/*
# Install Apache Spark
# https://downloads.apache.org/spark/KEYS
ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.3.2/spark-3.3.2-bin-hadoop3.tgz \
SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.3.2/spark-3.3.2-bin-hadoop3.tgz.asc \
GPG_KEY=C56349D886F2B01F8CAE794C653C2301FEA493EE
RUN set -ex; \
export SPARK_TMP="$(mktemp -d)"; \
cd $SPARK_TMP; \
wget -nv -O spark.tgz "$SPARK_TGZ_URL"; \
wget -nv -O spark.tgz.asc "$SPARK_TGZ_ASC_URL"; \
export GNUPGHOME="$(mktemp -d)"; \
gpg --keyserver hkps://keys.openpgp.org --recv-key "$GPG_KEY" || \
gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys "$GPG_KEY"; \
gpg --batch --verify spark.tgz.asc spark.tgz; \
gpgconf --kill all; \
rm -rf "$GNUPGHOME" spark.tgz.asc; \
\
tar -xf spark.tgz --strip-components=1; \
chown -R spark:spark .; \
mv jars /opt/spark/; \
mv bin /opt/spark/; \
mv sbin /opt/spark/; \
mv kubernetes/dockerfiles/spark/decom.sh /opt/; \
mv examples /opt/spark/; \
mv kubernetes/tests /opt/spark/; \
mv data /opt/spark/; \
mv python/pyspark /opt/spark/python/pyspark/; \
mv python/lib /opt/spark/python/lib/; \
cd ..; \
rm -rf "$SPARK_TMP"; \
ln -s /usr/share/java/postgresql-jdbc4.jar /opt/spark/jars/postgresql-jdbc4.jar;
COPY entrypoint.sh /opt/
ENV SPARK_HOME /opt/spark
ENV PATH="$PATH:/opt/spark/bin"
WORKDIR /opt/spark/work-dir
RUN chmod g+w /opt/spark/work-dir
RUN chmod a+x /opt/decom.sh
RUN chmod a+x /opt/entrypoint.sh
ENTRYPOINT [ "/opt/entrypoint.sh" ]
Contributor guide
Assessment
This issue has not been assessed yet.