[BUG] Not found exception for class in org.apache.log4j.*
- Dominant language
- Java
- Stars
- 399
- Forks
- 80
- Avg merge
- 3d 23h
- Merged PRs (30d)
- 15
Description
### Willingness to contribute
Yes. I would be willing to contribute a fix for this bug with guidance from the OpenHouse community.
### OpenHouse version
0.5.116
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 20.0)**: Ubuntu 24.04 LTS
- **JDK version**: openjdk version "1.8.0_422"
### Describe the problem
While trying to start Spark 3.1.1, e.g., the thrift server, we get a `ClassNotFoundException` (stacktrace below). It seems that the OpenHouse packaging does not rename `org.apache.log4j` dependencies in `apps/spark/build.gradle`, but there might be some constant references that cause the issue.
```
starting org.apache.spark.sql.hive.thriftserver.HiveThriftServer2, logging to /home/ohuser/spark/logs/spark-ohuser-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-oh-headnode.out
failed to launch: nice -n 0 bash /home/ohuser/spark/bin/spark-submit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name Thrift JDBC/ODBC Server --packages org.apache.iceberg:iceberg-azure:1.5.0,org.apache.iceberg:iceberg-spark-runtime-3.1_2.12:1.2.0 --jars openhouse-spark-apps_2.12-*-all.jar,openhouse-spark-runtime_2.12-latest-all.jar --conf spark.sql.catalog.openhouse=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.openhouse.catalog-impl=com.linkedin.openhouse.spark.OpenHouseCatalog --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,com.linkedin.openhouse.spark.extensions.OpenhouseSparkSessionExtensions
at org.apache.spark.internal.Logging.initializeLogIfNecessary$(Logging.scala:105)
at org.apache.spark.deploy.SparkSubmit.initializeLogIfNecessary(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:83)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: openhouse/relocated/org.apache.log4j.Category
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
... 15 more
full log in /home/ohuser/spark/logs/spark-ohuser-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-oh-headnode.out
```
### Stacktrace, metrics and logs
_No response_
### Code to reproduce bug
_No response_
### What component does this bug affect?
- [ ] `Table Service`: This is the RESTful catalog service that stores table metadata. `:services:tables`
- [ ] `Jobs Service`: This is the job orchestrator that submits data services for table maintenance. `:services:jobs`
- [X] `Data Services`: This is the jobs that performs table maintenance. `apps:spark`
- [ ] `Iceberg internal catalog`: This is the internal Iceberg catalog for OpenHouse Catalog Service. `:iceberg:openhouse`
- [ ] `Spark Client Integration`: This is the Apache Spark integration for OpenHouse catalog. `:integration:spark`
- [ ] `Documentation`: This is the documentation for OpenHouse. `docs`
- [ ] `Local Docker`: This is the local Docker environment for OpenHouse. `infra/recipes/docker-compose`
- [ ] `Other`: Please specify the component.
Contributor guide
Research direction
Inspect apps/spark/build.gradle first, focusing on dependency relocation for org.apache.log4j. Reproduce the failure by starting Spark 3.1.1's HiveThriftServer2 with spark-submit and the listed packages and jars. Done means the thrift server starts without ClassNotFoundException for openhouse/relocated/org.apache.log4j.Category.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100