linkedin / linkedin/openhouse

[BUG] Local docker failed to run spark-shell on Mac M1

Open
#115 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
399
Forks
80
Avg merge
3d 23h
Merged PRs (30d)
15

Description

### Willingness to contribute

Yes. I can contribute a fix for this bug independently.

### OpenHouse version

v0.5.62

### System information

- **OS Platform and Distribution (e.g., Linux Ubuntu 20.0)**: Apple M1, MacOS Sonoma 14.5, Docker v4.30
- **JDK version**: 1.8
###

### Describe the problem

While running `spark-shell` commands in the `SETUP.md`, it always prompts a fatal error related to Java Runtime Environment.

After investigation, I found that it is a common docker issue on Apple Silicon Macbook due to a bug in Rosseta (the x86/amd64 emulation application on Apple Silicon).

More details about this issue can be found in [https://github.com/docker/for-mac/issues/7006](url)

While waiting for the fix from Apple, there are several workarounds for this issue. For me, downgrading the Docker to [https://docs.docker.com/desktop/release-notes/#4272](version 4.27.2) will work. Additionally, other methods are mentioned in [https://github.com/docker/for-mac/issues/7006#issuecomment-2122869966](other).

### Stacktrace, metrics and logs

```bash
A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007ffffe0b8e1e, pid=692, tid=0x00007fffe86e6700
#
# JRE version: OpenJDK Runtime Environment (8.0_232-b09) (build 1.8.0_232-8u232-b09-1~deb9u1-b09)
# Java VM: OpenJDK 64-Bit Server VM (25.232-b09 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V [libjvm.so+0x628e1e]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /opt/spark/hs_err_pid692.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
```

### Code to reproduce bug

```bash
bin/spark-shell --packages org.apache.iceberg:iceberg-spark-runtime-3.1_2.12:1.2.0 \
--jars openhouse-spark-runtime_2.12-*-all.jar \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,com.linkedin.openhouse.spark.extensions.OpenhouseSparkSessionExtensions \
--conf spark.sql.catalog.openhouse=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.openhouse.catalog-impl=com.linkedin.openhouse.spark.OpenHouseCatalog \
--conf spark.sql.catalog.openhouse.metrics-reporter-impl=com.linkedin.openhouse.javaclient.OpenHouseMetricsReporter \
--conf spark.sql.catalog.openhouse.uri=http://openhouse-tables:8080 \
--conf spark.sql.catalog.openhouse.auth-token=$(cat /var/config/$(whoami).token) \
--conf spark.sql.catalog.openhouse.cluster=LocalHadoopCluster
```

### What component does this bug affect?

- [ ] `Table Service`: This is the RESTful catalog service that stores table metadata. `:services:tables`
- [ ] `Jobs Service`: This is the job orchestrator that submits data services for table maintenance. `:services:jobs`
- [ ] `Data Services`: This is the jobs that performs table maintenance. `apps:spark`
- [ ] `Iceberg internal catalog`: This is the internal Iceberg catalog for OpenHouse Catalog Service. `:iceberg:openhouse`
- [ ] `Spark Client Integration`: This is the Apache Spark integration for OpenHouse catalog. `:integration:spark`
- [ ] `Documentation`: This is the documentation for OpenHouse. `docs`
- [X] `Local Docker`: This is the local Docker environment for OpenHouse. `infra/recipes/docker-compose`
- [ ] `Other`: Please specify the component.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the `spark-shell` command from `SETUP.md` on Apple Silicon and inspect the local Docker setup under `infra/recipes/docker-compose`. Determine whether the project can avoid the reported Docker/Rosetta failure or needs a documented workaround, then verify that the setup produces a usable `spark-shell` environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, java, macos, spark
Domain
cli, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.