Feature request: hermetic build
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
When I was building datafusion for the first time (with `cargo test`), I met an error:
```
error: failed to run custom build command for `substrait v0.53.2`
Caused by:
process didn't exit successfully: `/home/ubuntu/datafusion/target/debug/build/substrait-79161c402651bf24/build-script-build` (exit status: 1)
--- stdout
cargo:rerun-if-env-changed=FORCE_REBUILD
cargo:rerun-if-changed=substrait
cargo:rerun-if-changed=substrait/text/simple_extensions_schema.yaml
cargo:rerun-if-changed=substrait/proto/substrait/parameterized_types.proto
cargo:rerun-if-changed=substrait/proto/substrait/type.proto
cargo:rerun-if-changed=substrait/proto/substrait/plan.proto
cargo:rerun-if-changed=substrait/proto/substrait/extensions/extensions.proto
cargo:rerun-if-changed=substrait/proto/substrait/function.proto
cargo:rerun-if-changed=substrait/proto/substrait/extended_expression.proto
cargo:rerun-if-changed=substrait/proto/substrait/capabilities.proto
cargo:rerun-if-changed=substrait/proto/substrait/algebra.proto
cargo:rerun-if-changed=substrait/proto/substrait/type_expressions.proto
--- stderr
Error: Custom { kind: NotFound, error: "Could not find `protoc`. If `protoc` is installed, try setting the `PROTOC` environment variable to the path of the `protoc` binary. To install it on Debian, run `apt-get install protobuf-compiler`. It is also available at https://github.com/protocolbuffers/protobuf/releases For more information: https://docs.rs/prost-build/#sourcing-protoc" }
```
which means it requires `protoc` installed on local as a pre-requisite.
It would be nice if we could support [hermetic build](https://bazel.build/basics/hermeticity), so dependencies for datafusion doesn't rely on system, unless necessary (i.e. glibc).
(I'm fairly new to rust and datafusion, might ask a dummy question.)
### Describe the solution you'd like
The way [bazel](https://bazel.build/) achieves hermeticity is
(1 ) include every dependency (with their version pinned) in the repository,
(2) download in source at compilation, and they could be modified if necessary via patch
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by reproducing the reported `cargo test` failure and inspect the `substrait` custom build command that requires `protoc`. Review the repository's current dependency and build configuration before deciding how hermeticity should be scoped. Done would mean a documented, agreed approach that avoids relying on an uninstalled system `protoc` while preserving the existing build.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- build-system
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100