[VL] Document CA bundle path fix for S3 hang on non-RHEL runtime images (root cause of #10670)
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
## Summary
This is a follow-up to #10670. We independently hit the exact same symptom (Gluten native S3 access hangs with no error, works fine when Gluten is disabled) on our own environment, and were able to identify the root cause and a reproducible fix. Since #10670 was closed without a concrete resolution, and the same issue is very likely to keep recurring for anyone running official/nightly Gluten binaries on a non-RHEL-family container image, we'd like to share the findings here and propose a small documentation improvement.
## Environment where the issue was reproduced
- Gluten versions: 1.6.0, 1.7.0-SNAPSHOT (both official/nightly binaries, no custom build)
- Spark: 4.0.x, Velox backend
- Runtime container base image: Ubuntu (via `apache/spark` official Docker image with `-ubuntu` tag)
- Storage: S3-compatible object storage (non-AWS, MinIO-based)
- Symptom: with `spark.gluten.sql.columnar.batchscan=true`, queries hang indefinitely with no error/exception in driver or executor logs. Disabling Gluten (`spark.gluten.enabled=false`) makes the exact same query complete normally.
## Root cause
Gluten's native library (`libvelox.so`) is compiled on CentOS/Rocky Linux as part of the official build/release pipeline. The native AWS SDK (used via curl) that Velox links against resolves the CA bundle from the RHEL-family standard path `/etc/pki/tls/certs/ca-bundle.crt`, which is hardcoded at build/link time.
On a non-RHEL-family runtime (Ubuntu/Debian, where the CA bundle instead lives at `/etc/ssl/certs/ca-certificates.crt`), this path does not exist. The native curl call then fails the TLS handshake (`curlCode: 77`), and an internal exponential-backoff retry loop kicks in — from the Spark driver/executor's point of view this looks exactly like a silent hang, with no exception surfaced to Spark/JVM logs.
This matches the disclaimer on the official [Downloads page](https://gluten.apache.org/downloads/):
> Please note that the binary files were compiled on CentOS 7 using static linking... Performance and runtime compatibility are not guaranteed on other operating systems or hardware platforms.
We confirmed:
- The problem reproduces identically regardless of which Gluten jar is used (official release build, nightly build, or a jar built via the project's own CentOS/Rocky-based CI pipeline) — the jar's origin doesn't matter, since all of them are compiled on the same OS family.
- Setting standard environment variables such as `SSL_CERT_FILE` or `CURL_CA_BUNDLE` has **no effect** — these are not honored by the native curl call inside the Velox/AWS SDK layer. This is a common first thing people try (understandably, since it works for many other tools), so it's worth explicitly documenting that it does *not* work here.
## Fix
No Gluten source code or jar rebuild is required. Adding a single symlink at container image build time (as root) is sufficient:
```dockerfile
RUN mkdir -p /etc/pki/tls/certs && \
ln -sf /etc/ssl/certs/ca-certificates.crt /etc/pki/tls/certs/ca-bundle.crt
```
After applying this, `batchscan=true` (native scan) works correctly with no hang, and we further confirmed it is consistently faster than the `batchscan=false` (JVM fallback) workaround across multiple query patterns and submission methods — typically in the range of roughly 1.3x–2.2x, with identical query results in all cases.
## Proposed enhancement
Since this isn't something Gluten can unilaterally "fix" for every possible runtime OS (as the Downloads page already acknowledges), we'd like to propose adding a short note to the relevant getting-started documentation (e.g. `docs/get-started/VeloxS3.md` or similar), along the lines of:
> If you deploy the official Gluten binaries on a non-RHEL-family base image (e.g. Ubuntu/Debian), native S3 access may silently hang due to a hardcoded CA bundle path expectation. Symlinking your distribution's CA bundle to `/etc/pki/tls/certs/ca-bundle.crt` resolves this. Note that `SSL_CERT_FILE`/`CURL_CA_BUNDLE` environment variables are not effective for this.
This would save future users (especially anyone deploying on Debian/Ubuntu based Spark images, which are quite common) significant debugging time, as the current symptom (silent hang, no errors) makes this very hard to diagnose from first principles — as evidenced by #10670 remaining unresolved for months.
Happy to help draft the doc PR if this is welcome.
---
*This issue was written with the assistance of AI (used to help organize and phrase the investigation notes).*
### Gluten version
main branch
Contributor guide
Research direction
Start with docs/get-started/VeloxS3.md and the Downloads page disclaimer to locate the relevant S3 deployment guidance. Add a concise note covering non-RHEL CA bundle paths, the symlink workaround, and ineffective environment variables; done means the documented symptom and fix are easy to find and understand.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, ubuntu
- Domain
- cloud, documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 75/100