linkedin / linkedin/openhouse

[BUG] Not found exception for class in org.apache.log4j.*

Open
#183 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
399
Forks
80
Avg merge
3d 23h
Merged PRs (30d)
15

Description

### Willingness to contribute

Yes. I would be willing to contribute a fix for this bug with guidance from the OpenHouse community.

### OpenHouse version

0.5.116

### System information

- **OS Platform and Distribution (e.g., Linux Ubuntu 20.0)**: Ubuntu 24.04 LTS
- **JDK version**: openjdk version "1.8.0_422"

### Describe the problem

While trying to start Spark 3.1.1, e.g., the thrift server, we get a `ClassNotFoundException` (stacktrace below). It seems that the OpenHouse packaging does not rename `org.apache.log4j` dependencies in `apps/spark/build.gradle`, but there might be some constant references that cause the issue.

```
starting org.apache.spark.sql.hive.thriftserver.HiveThriftServer2, logging to /home/ohuser/spark/logs/spark-ohuser-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-oh-headnode.out
failed to launch: nice -n 0 bash /home/ohuser/spark/bin/spark-submit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name Thrift JDBC/ODBC Server --packages org.apache.iceberg:iceberg-azure:1.5.0,org.apache.iceberg:iceberg-spark-runtime-3.1_2.12:1.2.0 --jars openhouse-spark-apps_2.12-*-all.jar,openhouse-spark-runtime_2.12-latest-all.jar --conf spark.sql.catalog.openhouse=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.openhouse.catalog-impl=com.linkedin.openhouse.spark.OpenHouseCatalog --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,com.linkedin.openhouse.spark.extensions.OpenhouseSparkSessionExtensions
at org.apache.spark.internal.Logging.initializeLogIfNecessary$(Logging.scala:105)
at org.apache.spark.deploy.SparkSubmit.initializeLogIfNecessary(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:83)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: openhouse/relocated/org.apache.log4j.Category
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
... 15 more
full log in /home/ohuser/spark/logs/spark-ohuser-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-oh-headnode.out
```

### Stacktrace, metrics and logs

_No response_

### Code to reproduce bug

_No response_

### What component does this bug affect?

- [ ] `Table Service`: This is the RESTful catalog service that stores table metadata. `:services:tables`
- [ ] `Jobs Service`: This is the job orchestrator that submits data services for table maintenance. `:services:jobs`
- [X] `Data Services`: This is the jobs that performs table maintenance. `apps:spark`
- [ ] `Iceberg internal catalog`: This is the internal Iceberg catalog for OpenHouse Catalog Service. `:iceberg:openhouse`
- [ ] `Spark Client Integration`: This is the Apache Spark integration for OpenHouse catalog. `:integration:spark`
- [ ] `Documentation`: This is the documentation for OpenHouse. `docs`
- [ ] `Local Docker`: This is the local Docker environment for OpenHouse. `infra/recipes/docker-compose`
- [ ] `Other`: Please specify the component.

Contributor guide

Open the contributing guide

Research direction

Inspect apps/spark/build.gradle first, focusing on dependency relocation for org.apache.log4j. Reproduce the failure by starting Spark 3.1.1's HiveThriftServer2 with spark-submit and the listed packages and jars. Done means the thrift server starts without ClassNotFoundException for openhouse/relocated/org.apache.log4j.Category.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.