Blizzard / Blizzard/node-rdkafka
Option to Skip Rebuild in CI Environments
- Dominant language
- JavaScript
- Stars
- 2.2k
- Forks
- 403
- PR merge metrics
- No merged PRs in 30d
Description
## Description
In our project, we use `node-rdkafka` in a CI environment where we rely on `npm ci --cache ...` to ensure clean installations.
However, because `npm ci` removes the entire `node_modules` folder on every run, node-rdkafka’s default behavior of executing `node-gyp rebuild` always triggers a full rebuild of its C++ components.
This significantly increases our CI build times. In our scale, we have hundreds of CI workers run per day, and the increase in time is unacceptable. Also, it's just wasted resources, as all workers need the same binaries in 99.999% of cases.
We use `npm` cache, but the build binaries are not cached by npm, only base package files.
We currently maintain a [dedicated fork](https://github.com/codequest-eu/node-rdkafka) that provides prebuilt binaries to avoid this rebuild step, but this approach is cumbersome due to the ongoing need to sync with the main `node-rdkafka` repository (and other reasons).
## Problem
- **CI Build Overhead:** Every CI run triggers `node-gyp rebuild` because the build artifacts are not preserved, resulting in longer build times.
- **Fork Maintenance:** Maintaining a fork solely to provide prebuilt binaries introduces additional maintenance overhead and drift risk with the upstream repository.
- **Lack of Bypass Mechanism:** There is currently no built-in option in node-rdkafka (or node-gyp) to skip the rebuild when a valid prebuilt binary is available.
## Proposed Solution
**Environment Variable**
Introduce an optional environment variable (e.g., `NODE_RDKAFKA_SKIP_BUILD`) that, when set, would check for an available prebuilt binary and skip the `node-gyp rebuild` step if found. This would allow users running in CI or other environments to maintain the cache manually (e.g. in onw package `postinstall` script) and avoid unnecessary recompilation.
## Questions for the Maintainers
- Is there a recommended approach or best practice for handling this scenario in CI environments without resorting to maintaining a dedicated fork?
- Would you consider a pull request that implements an option (via environment variable) to skip the build?
I'm happy to contribute to a solution and help test any proposed changes.
Contributor guide
Research direction
No source files or tests are named. Start by tracing the installation path that invokes node-gyp rebuild, then determine how an environment variable could safely recognize a valid prebuilt binary and bypass rebuilding; done means the documented CI scenario avoids unnecessary recompilation without breaking normal installs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- build-system
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100