apache / apache/datafusion

`datafusion-cli` requires more than 8GB to build

Open
#18,471 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

@pmcgleenon reports on https://github.com/apache/datafusion/issues/17721#issuecomment-3476400608

> The instance type [c6a.xlarge](https://instances.vantage.sh/aws/ec2/c6a.xlarge) only has 8GB RAM (the clickbench dataset is 15GB) and there are a number of issues with it, including
>
> datafusion compilation fails on this instance. We can workaround this by using brew install

### To Reproduce

You can reproduce this locally on docker using this command:

```shell
cd .devcontainer
# create a docker container
docker build -t datafusion-build .
cd ..
docker run -m 4G -v `pwd`:/datafusion -it datafusion-build /bin/bash
```

Then try to run
```
cargo build --release -p datafusion-cli
```

This will fail with an error like this:

```
Caused by:
process didn't exit successfully: `/usr/local/rustup/toolchains/1.90.0-aarch64-unknown-linux-gnu/bin/rustc --crate-name datafusion_functions_aggregate --edition=2021 datafusion/functions-aggregate/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=160 --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C linker-plugin-lto -C codegen-units=1 '--warn=clippy::used_underscore_binding' --deny=unused_qualifications '--warn=clippy::unnecessary_lazy_evaluations' '--warn=clippy::uninlined_format_args' --warn=unexpected_cfgs '--warn=clippy::or_fun_call' '--warn=clippy::large_futures' '--warn=clippy::inefficient_to_string' --check-cfg 'cfg(datafusion_coop, values("tokio", "tokio_fallback", "per_stream"))' --check-cfg 'cfg(tarpaulin)' --check-cfg 'cfg(tarpaulin_include)' --check-cfg 'cfg(docsrs,test)' --check-cfg 'cfg(feature, values())' -C metadata=727ec34a83a3b23a -C extra-filename=-41481de704778bc6 --out-dir /datafusion/target/release/deps -C strip=symbols -L dependency=/datafusion/target/release/deps --extern ahash=/datafusion/target/release/deps/libahash-16bfcefaf019ce0a.rmeta --extern arrow=/datafusion/target/release/deps/libarrow-7a0c4e030b71b835.rmeta --extern datafusion_common=/datafusion/target/release/deps/libdatafusion_common-ca5a073002b6fd0e.rmeta --extern datafusion_doc=/datafusion/target/release/deps/libdatafusion_doc-8ab038afb63bc362.rmeta --extern datafusion_execution=/datafusion/target/release/deps/libdatafusion_execution-93c2cc956c650df0.rmeta --extern datafusion_expr=/datafusion/target/release/deps/libdatafusion_expr-87611f1ca5ce8f65.rmeta --extern datafusion_functions_aggregate_common=/datafusion/target/release/deps/libdatafusion_functions_aggregate_common-05b65785638f4212.rmeta --extern datafusion_macros=/datafusion/target/release/deps/libdatafusion_macros-0bed692e75dab347.so --extern datafusion_physical_expr=/datafusion/target/release/deps/libdatafusion_physical_expr-93e2a25460efd9b7.rmeta --extern datafusion_physical_expr_common=/datafusion/target/release/deps/libdatafusion_physical_expr_common-79bb18eb023feafb.rmeta --extern half=/datafusion/target/release/deps/libhalf-feba467d2af9eeb5.rmeta --extern log=/datafusion/target/release/deps/liblog-d456af582e58c114.rmeta --extern paste=/datafusion/target/release/deps/libpaste-5dba9edad0280e64.so -L native=/datafusion/target/release/build/zstd-sys-85e0b1f1ab4f39ce/out -L native=/datafusion/target/release/build/lzma-sys-d68250324f9032a4/out -L native=/datafusion/target/release/build/aws-lc-sys-441c9757ee7585c9/out -L native=/datafusion/target/release/build/ring-dcbccc510e078148/out -L native=/datafusion/target/release/build/psm-2382bc49ac467ad4/out` (signal: 9, SIGKILL: kill)
```

Note the `signal: 9, SIGKILL: kill`

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the .devcontainer setup and reproduce the failure using `docker build`, then run `cargo build --release -p datafusion-cli` inside the 4G container. Investigate the reported SIGKILL during Rust compilation; done means the datafusion-cli release build completes within the constrained memory environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, rust
Domain
build-system, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.