aboutcode-org / aboutcode-org/scancode.io

ignored_dependency_scopes does not exclude npm devDependencies

Đang mở
#2,215 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
215
Fork
203
Merge trung bình
4 ngày 8 giờ
Pull request đã merge (30 ngày)
6

Mô tả

### Description

The `ignored_dependency_scopes` setting in `scancode-config.yml` does not appear to exclude npm `devDependencies` during a ScanCode.io scan.

I encountered this while scanning ScanCode Workbench 4.0.2. I intentionally used the same configuration that is documented in the official ScanCode.io end-to-end tutorial:

```yaml
product_name: scancode-workbench
product_version: '4.0.2'
ignored_patterns:
- 'tests/*'
ignored_dependency_scopes:
- package_type: npm
scope: devDependencies
```

The `scancode-config.yml` is located at the root of the project inside a single uploaded source archive.

The scan completes successfully, but dependencies with the scope `devDependencies` are still included instead of being ignored.

This seems to affect the `ignored_dependency_scopes` filtering itself rather than npm dependency detection, since ScanCode Toolkit correctly identifies these dependencies with the scope `devDependencies`.

### How To Reproduce

Steps:

1. Use [ScanCode Workbench 4.0.2](https://github.com/aboutcode-org/scancode-workbench/archive/refs/tags/v4.0.2.tar.gz) as the npm project.
2. Add the following `scancode-config.yml` to the project root:

```yaml
product_name: scancode-workbench
product_version: '4.0.2'
ignored_patterns:
- 'tests/*'
ignored_dependency_scopes:
- package_type: npm
scope: devDependencies
```

3. Create a source archive containing the project and configuration.
4. Create a ScanCode.io project and upload the archive as a single input.
5. Run package inspection/scanning so that npm dependencies from `package-lock.json` are collected.
6. Inspect the resulting dependencies.

Dependencies with:

```text
scope: devDependencies
```

are still present even though they should be ignored by the project configuration.

The provided `package-lock.json` uses lockfile version 3 and contains a large number of dependencies marked as development dependencies, so the behavior is easy to reproduce.

### Expected behavior

npm dependencies with the scope `devDependencies` should be ignored when the following configuration is present:

```yaml
ignored_dependency_scopes:
- package_type: npm
scope: devDependencies
```

According to the ScanCode.io documentation, dependencies matching an ignored package type and scope should not be created or resolved.

This configuration is also the exact example currently used in the ScanCode.io tutorial with ScanCode Workbench 4.0.2.

### Actual behavior

npm dependencies with `scope: devDependencies` are still created/included in the scan results.

### Suspected root cause

I looked into the implementation and suspect that the issue is in `ignore_dependency_scope()` in:

`scanpipe/pipes/__init__.py`

The function currently retrieves the package type and scope from the dependency data:

```python
dependency_package_type = dependency_data.get("package_type")
dependency_scope = dependency_data.get("scope")
```

and only evaluates the ignore rule when both values are present.

However, ScanCode Toolkit's `DependentPackage` / `Dependency` model does not appear to contain a `package_type` field.

The dependency model contains fields such as:

```text
purl
extracted_requirement
scope
is_runtime
is_optional
is_pinned
is_direct
resolved_package
extra_data
```

For an npm dependency, the package type is already represented by the PURL, for example:

```text
pkg:npm/...
```

while the dependency scope is correctly represented as:

```text
devDependencies
```

Therefore, for a normal dependency generated by ScanCode Toolkit, this lookup appears to result in:

```python
dependency_data.get("package_type") # None
```

As a consequence, the following condition is not entered:

```python
if dependency_package_type and dependency_scope:
```

and the configured ignore rule is never evaluated.

### Relation to the original implementation

`ignored_dependency_scopes` was introduced as part of #1197 and implemented in #1235.

There is another interesting detail in the tests added with #1235: for the positive `ignore_dependency_scope()` test case, `package_type` appears to be manually added to the dependency dictionary before calling the function.

In other words, the test verifies the behavior with something similar to:

```python
dependency_data["package_type"] = "pypi"
dependency_data["scope"] = "tests"
```

This may explain why the implementation passes its unit test even though a normal dependency produced by ScanCode Toolkit does not contain that `package_type` field.

A regression test using an actual ScanCode Toolkit `Dependency` / `DependentPackage` object may expose the problem.

### Possible approach

One possible solution could be to derive the package type from the dependency PURL when `package_type` is not explicitly available.

For example, a dependency such as:

```text
pkg:npm/example@1.0.0
```

already provides the required package type (`npm`).

Alternatively, the dependency data could be normalized so that the package type is populated before `ignore_dependency_scope()` is called.

The exact implementation is of course up to the maintainers.

### Related issues / changes

* #1197 – Enhance and create detailed documentation for `scancode-config.yml`
* #1235 – Add support for `ignored_dependency_scopes` field for configuration
* #1236 – Support storing `scancode-config.yml` files in the scanned codebase

#1236 specifically documents that, for a single archive input, a `scancode-config.yml` located at the root should be used.

The official ScanCode.io end-to-end tutorial also currently uses ScanCode Workbench 4.0.2 together with:

```yaml
ignored_dependency_scopes:
- package_type: npm
scope: devDependencies
```

which makes this project a convenient reproducer.

### System configuration

* ScanCode.io: 37.2.0
* ScanCode Toolkit: 32.5.0
* Installation: Docker
* Package ecosystem: npm
* npm lockfile: `package-lock.json`, lockfile version 3
* Input: single source code archive containing `scancode-config.yml`

I also checked the current ScanCode.io `main` branch, and `ignore_dependency_scope()` still appears to use `dependency_data.get("package_type")` in the same way.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.