nodejs / nodejs/gyp-next

Makefile generator creates misplaced build artifacts for dependencies outside of `binding.gyp` root dir

Open
#325 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
155
Forks
101
Avg merge
6d 2h
Merged PRs (30d)
4

Description

A binding.gyp with dependencies that live outside of its directory will create misplaced artifacts with the makefile generator (usually, on mac & linux). The dependency will be resolved to a relative path with parent directory segments (../../*) and those get joined as-is to the build directory (./build/../../*), resulting in build files outside of the build directory. This result may confuse other tooling too, unfortunately.

For example, the following code in the makefile generator produces a path like ./build/../../node_modules/node-addon-api/node_addon_api.target.mk:

https://github.com/nodejs/gyp-next/blob/732b09edc711ce3bad2df6db02ba8d5f315ca375/pylib/gyp/generator/make.py#L2433-L2436

Motivating example

I have a multi-workspace project set up like this:

  • super/ (root)
    • package.json (defines "workspaces")
    • workspaces/
      • native/
        • package.json
        • binding.gyp

super has no dependencies, and native only depends on node-gyp and node-addon-api. Note that in this setup, npm hoists node-addon-gyp to the root node_modules/.

node-addon-api recommends adding it as a dependency in binding.gyp:

# super/workspaces/native/binding.gyp
{
    "targets": [{
        "target_name": "native",
        "sources": ["module.cc"],
        "dependencies": ["<!(node -p \"require('node-addon-api').targets\"):node_addon_api"]
    }]
}

After running node-gyp rebuild, we end up with some mysterious build artifacts:

  • super/ (root)
    • node_modules/ ... (real, created by npm)
    • workspaces/
      • node_modules/ (not real, created by gyp)
        • node-addon-api/
          • node_addon_api_except_all.target.mk
          • node_addon_api_except.target.mk
          • node_addon_api_maybe.target.mk
          • node_addon_api.target.mk
          • node_addon_api.Makefile

A real issue that this causes is npm list failing:

$ npm ls

super@0.0.1 /[...]/super
├─┬ native@0.0.1 -> ./workspaces/native
│ ├── bindings@1.5.0
│ ├── node-addon-api@ invalid: "^8.5.0" from workspaces/native
│ └── node-gyp@12.1.0
└── node-addon-api@8.5.0 extraneous

npm error code ELSPROBLEMS
npm error extraneous: node-addon-api@8.5.0 /[...]/node_modules/node-addon-api
npm error invalid: node-addon-api@ /[...]/workspaces/node_modules/node-addon-api

# exit code 1

We also see some misplaced files within the build directory too:

  • super/workspaces/native/build/
    • node_modules/ (should be within Release/obj.target/)
      • node-addon-api/
        • node_addon_api.stamp
    • Release/ ...

I haven't found these artifacts to cause issues, but I am highlighting them because they share the same root cause.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pylib/gyp/generator/make.py at the dependency path handling around lines 2433-2436, then reproduce the issue with the described multi-workspace binding.gyp and node-gyp rebuild setup. Done means dependencies outside the binding.gyp directory no longer create makefiles, stamps, or other artifacts outside the intended build locations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.