apache / apache/datafusion

suggest compile binaries with dynamic libs

Open
#16,600 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

when I compile benchmarks binaries with `cargo build --release` and use `mold` as the linker, it occurs following errors
```
(signal: 9, SIGKILL: kill)
warning: build failed, waiting for other jobs to finish...
error: could not compile `datafusion-benchmarks` (bin "dfbench")
...
(signal: 9, SIGKILL: kill)
Building [=======================> ] 443/446: imdb(bin), tpch(bin), external_aggr(bin)
```
and the errors disappeared when I run `CARGO_BUILD_JOBS=1 cargo build --release`
but each binary took about 5 to 10 min to compile, it's too slow.
```
Finished `release` profile [optimized] target(s) in 25m 01s
```
I found the binaries share many rlib files, such as libdatafusion_benchmarks-bb94f26d75d1c935.rlib, could you compile them into .so files, and let the smaller binary load the .so files at runtime? it can save many time and CPU.

### Describe the solution you'd like

compile them into .so files, and let the smaller binary load the .so files at runtime

### Describe alternatives you've considered

supply the binaries when publish the new release

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the `datafusion-benchmarks` release build with `cargo build --release` and `mold`, comparing parallel compilation with `CARGO_BUILD_JOBS=1`. Investigate how the benchmark binaries share rlib files and define a build approach that produces reusable dynamic libraries without breaking runtime loading; done means the release build avoids repeated compilation while the binaries still run correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.