[BUG] Local docker failed to run spark-shell on Mac M1
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 399
- Forks
- 80
- Avg merge
- 3d 23h
- Merged PRs (30d)
- 15
Description
### Willingness to contribute
Yes. I can contribute a fix for this bug independently.
### OpenHouse version
v0.5.62
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 20.0)**: Apple M1, MacOS Sonoma 14.5, Docker v4.30
- **JDK version**: 1.8
###
### Describe the problem
While running `spark-shell` commands in the `SETUP.md`, it always prompts a fatal error related to Java Runtime Environment.
After investigation, I found that it is a common docker issue on Apple Silicon Macbook due to a bug in Rosseta (the x86/amd64 emulation application on Apple Silicon).
More details about this issue can be found in [https://github.com/docker/for-mac/issues/7006](url)
While waiting for the fix from Apple, there are several workarounds for this issue. For me, downgrading the Docker to [https://docs.docker.com/desktop/release-notes/#4272](version 4.27.2) will work. Additionally, other methods are mentioned in [https://github.com/docker/for-mac/issues/7006#issuecomment-2122869966](other).
### Stacktrace, metrics and logs
```bash
A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007ffffe0b8e1e, pid=692, tid=0x00007fffe86e6700
#
# JRE version: OpenJDK Runtime Environment (8.0_232-b09) (build 1.8.0_232-8u232-b09-1~deb9u1-b09)
# Java VM: OpenJDK 64-Bit Server VM (25.232-b09 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V [libjvm.so+0x628e1e]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /opt/spark/hs_err_pid692.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
```
### Code to reproduce bug
```bash
bin/spark-shell --packages org.apache.iceberg:iceberg-spark-runtime-3.1_2.12:1.2.0 \
--jars openhouse-spark-runtime_2.12-*-all.jar \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,com.linkedin.openhouse.spark.extensions.OpenhouseSparkSessionExtensions \
--conf spark.sql.catalog.openhouse=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.openhouse.catalog-impl=com.linkedin.openhouse.spark.OpenHouseCatalog \
--conf spark.sql.catalog.openhouse.metrics-reporter-impl=com.linkedin.openhouse.javaclient.OpenHouseMetricsReporter \
--conf spark.sql.catalog.openhouse.uri=http://openhouse-tables:8080 \
--conf spark.sql.catalog.openhouse.auth-token=$(cat /var/config/$(whoami).token) \
--conf spark.sql.catalog.openhouse.cluster=LocalHadoopCluster
```
### What component does this bug affect?
- [ ] `Table Service`: This is the RESTful catalog service that stores table metadata. `:services:tables`
- [ ] `Jobs Service`: This is the job orchestrator that submits data services for table maintenance. `:services:jobs`
- [ ] `Data Services`: This is the jobs that performs table maintenance. `apps:spark`
- [ ] `Iceberg internal catalog`: This is the internal Iceberg catalog for OpenHouse Catalog Service. `:iceberg:openhouse`
- [ ] `Spark Client Integration`: This is the Apache Spark integration for OpenHouse catalog. `:integration:spark`
- [ ] `Documentation`: This is the documentation for OpenHouse. `docs`
- [X] `Local Docker`: This is the local Docker environment for OpenHouse. `infra/recipes/docker-compose`
- [ ] `Other`: Please specify the component.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the `spark-shell` command from `SETUP.md` on Apple Silicon and inspect the local Docker setup under `infra/recipes/docker-compose`. Determine whether the project can avoid the reported Docker/Rosetta failure or needs a documented workaround, then verify that the setup produces a usable `spark-shell` environment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, docker-compose, java, macos, spark
- Domain
- cli, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100