aboutcode-org / aboutcode-org/scancode-toolkit

Scanning files of type mjs.map might produce huge result files with useless matched_text information

オープン
#3,810 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
bug
主要言語
Python
スター
2.6k
フォーク
791
平均マージ
1日 12時間
マージ済み PR(30日)
5

説明

### Description

Use case: We are scanning NPM package archives with scancode using the option "--license-text" and generating a json result file.

For packages which contain files of type `mjs.map` we encounter huge result files. This seems to be due to the following facts:
- `mjs.map` files consist only of a single (long) line. Scancode does not do any special handling here (as it seems to do for `js.map` files)
- Scancode outputs the "whole_line" as matched_text. Probably the internal flag "has_long_lines" does not get set in this case.

We reproduced this behavior on different platforms:
- Ubuntu native (see instructions below), Windows 2019 Server native, Docker on Ubuntu
- Python 3.8 and Python 3.12
- Scancode 31 and 32

**Note:** Actually we have some environment (Docker Container with scancode based on eclipse-temurin base image) which - when run on Windows in Rancher Desktop (WSL2 using dockerd/moby as runtime) does not show this behavior.
Instead in this environment the `matched_text` only shows the match itself (probably flag has_long_lines get set here differently).
Notably when the same image is run within Docker on Ubuntu (using containerd as runtime) the above stated problem (huge files) also exists for this image.
We have not investigated whether this difference in behavior is due to host platform differences (Ubuntu vs. WSL2) or container runtime differences (containerd vs. dockerd) - or some other cause
Our current assumption is that the observed differences are originating somehow from libmagic (and underlying libraries).

**Whether the observed behavior is a bug or a (non) feature is unclear: At least it results in huge result files (increased here by factor 100 - typically tens of megabytes in size) and thus might result in nonfunctional issues. The matched_text data in any case is useless.**

### How To Reproduce

The following gives the commands on reproducing the observed effect based on a fresh Ubuntu 24.04 machine (including setup of Scancode).
The file `animations.mjs.map` from NPM package pkg:npm/%40angular/animations@15.2.10 is the file we scan for testing purposes.

```
sudo apt-get update && sudo apt-get upgrade
sudo add-apt-repository ppa:deadsnakes/ppa --yes
sudo apt-get install python3.8 python3.8-distutils
sudo apt install python-dev-is-python3 bzip2 xz-utils zlib1g libxml2-dev libxslt1-dev libpopt0
sudo apt-get install libgomp1

wget https://github.com/nexB/scancode-toolkit/releases/download/v32.1.0/scancode-toolkit-v32.1.0_py3.8-linux.tar.gz
tar -xvf scancode-toolkit-v32.1.0_py3.8-linux.tar.gz
cd scancode-toolkit-v32.1.0
./scancode --help
cd ..

curl -o animations.tgz https://registry.npmjs.org/@angular/animations/-/animations-15.2.10.tgz
tar xvf animations.tgz

./scancode-toolkit-v32.1.0/scancode -clipeu --license-text --json-pp scan-result.json package/fesm2015/animations.mjs.map
less scan-result.json

```

### System configuration

See infos above.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。