aboutcode-org / aboutcode-org/scancode-toolkit

Scanning files of type mjs.map might produce huge result files with useless matched_text information

未關閉
#3,810 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
bug
主要語言
Python
星號
2.6k
分支
791
平均合併
1 天 12 小時
30 天內合併 PR
5

描述

### Description

Use case: We are scanning NPM package archives with scancode using the option "--license-text" and generating a json result file.

For packages which contain files of type `mjs.map` we encounter huge result files. This seems to be due to the following facts:
- `mjs.map` files consist only of a single (long) line. Scancode does not do any special handling here (as it seems to do for `js.map` files)
- Scancode outputs the "whole_line" as matched_text. Probably the internal flag "has_long_lines" does not get set in this case.

We reproduced this behavior on different platforms:
- Ubuntu native (see instructions below), Windows 2019 Server native, Docker on Ubuntu
- Python 3.8 and Python 3.12
- Scancode 31 and 32

**Note:** Actually we have some environment (Docker Container with scancode based on eclipse-temurin base image) which - when run on Windows in Rancher Desktop (WSL2 using dockerd/moby as runtime) does not show this behavior.
Instead in this environment the `matched_text` only shows the match itself (probably flag has_long_lines get set here differently).
Notably when the same image is run within Docker on Ubuntu (using containerd as runtime) the above stated problem (huge files) also exists for this image.
We have not investigated whether this difference in behavior is due to host platform differences (Ubuntu vs. WSL2) or container runtime differences (containerd vs. dockerd) - or some other cause
Our current assumption is that the observed differences are originating somehow from libmagic (and underlying libraries).

**Whether the observed behavior is a bug or a (non) feature is unclear: At least it results in huge result files (increased here by factor 100 - typically tens of megabytes in size) and thus might result in nonfunctional issues. The matched_text data in any case is useless.**

### How To Reproduce

The following gives the commands on reproducing the observed effect based on a fresh Ubuntu 24.04 machine (including setup of Scancode).
The file `animations.mjs.map` from NPM package pkg:npm/%40angular/animations@15.2.10 is the file we scan for testing purposes.

```
sudo apt-get update && sudo apt-get upgrade
sudo add-apt-repository ppa:deadsnakes/ppa --yes
sudo apt-get install python3.8 python3.8-distutils
sudo apt install python-dev-is-python3 bzip2 xz-utils zlib1g libxml2-dev libxslt1-dev libpopt0
sudo apt-get install libgomp1

wget https://github.com/nexB/scancode-toolkit/releases/download/v32.1.0/scancode-toolkit-v32.1.0_py3.8-linux.tar.gz
tar -xvf scancode-toolkit-v32.1.0_py3.8-linux.tar.gz
cd scancode-toolkit-v32.1.0
./scancode --help
cd ..

curl -o animations.tgz https://registry.npmjs.org/@angular/animations/-/animations-15.2.10.tgz
tar xvf animations.tgz

./scancode-toolkit-v32.1.0/scancode -clipeu --license-text --json-pp scan-result.json package/fesm2015/animations.mjs.map
less scan-result.json

```

### System configuration

See infos above.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。