apache / apache/datafusion

Feature request: hermetic build

Open
#14,678 3 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

When I was building datafusion for the first time (with `cargo test`), I met an error:
```
error: failed to run custom build command for `substrait v0.53.2`

Caused by:
process didn't exit successfully: `/home/ubuntu/datafusion/target/debug/build/substrait-79161c402651bf24/build-script-build` (exit status: 1)
--- stdout
cargo:rerun-if-env-changed=FORCE_REBUILD
cargo:rerun-if-changed=substrait
cargo:rerun-if-changed=substrait/text/simple_extensions_schema.yaml
cargo:rerun-if-changed=substrait/proto/substrait/parameterized_types.proto
cargo:rerun-if-changed=substrait/proto/substrait/type.proto
cargo:rerun-if-changed=substrait/proto/substrait/plan.proto
cargo:rerun-if-changed=substrait/proto/substrait/extensions/extensions.proto
cargo:rerun-if-changed=substrait/proto/substrait/function.proto
cargo:rerun-if-changed=substrait/proto/substrait/extended_expression.proto
cargo:rerun-if-changed=substrait/proto/substrait/capabilities.proto
cargo:rerun-if-changed=substrait/proto/substrait/algebra.proto
cargo:rerun-if-changed=substrait/proto/substrait/type_expressions.proto

--- stderr
Error: Custom { kind: NotFound, error: "Could not find `protoc`. If `protoc` is installed, try setting the `PROTOC` environment variable to the path of the `protoc` binary. To install it on Debian, run `apt-get install protobuf-compiler`. It is also available at https://github.com/protocolbuffers/protobuf/releases For more information: https://docs.rs/prost-build/#sourcing-protoc" }
```
which means it requires `protoc` installed on local as a pre-requisite.
It would be nice if we could support [hermetic build](https://bazel.build/basics/hermeticity), so dependencies for datafusion doesn't rely on system, unless necessary (i.e. glibc).

(I'm fairly new to rust and datafusion, might ask a dummy question.)

### Describe the solution you'd like

The way [bazel](https://bazel.build/) achieves hermeticity is
(1 ) include every dependency (with their version pinned) in the repository,
(2) download in source at compilation, and they could be modified if necessary via patch

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported `cargo test` failure and inspect the `substrait` custom build command that requires `protoc`. Review the repository's current dependency and build configuration before deciding how hermeticity should be scoped. Done would mean a documented, agreed approach that avoids relying on an uninstalled system `protoc` while preserving the existing build.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.