aboutcode-org / aboutcode-org/scancode-toolkit
Scanning files of type mjs.map might produce huge result files with useless matched_text information
- Lenguaje dominante
- Python
- Estrellas
- 2.6k
- Forks
- 791
- Merge medio
- 1 d 12 h
- PR fusionados (30 d)
- 5
Descripción
### Description
Use case: We are scanning NPM package archives with scancode using the option "--license-text" and generating a json result file.
For packages which contain files of type `mjs.map` we encounter huge result files. This seems to be due to the following facts:
- `mjs.map` files consist only of a single (long) line. Scancode does not do any special handling here (as it seems to do for `js.map` files)
- Scancode outputs the "whole_line" as matched_text. Probably the internal flag "has_long_lines" does not get set in this case.
We reproduced this behavior on different platforms:
- Ubuntu native (see instructions below), Windows 2019 Server native, Docker on Ubuntu
- Python 3.8 and Python 3.12
- Scancode 31 and 32
**Note:** Actually we have some environment (Docker Container with scancode based on eclipse-temurin base image) which - when run on Windows in Rancher Desktop (WSL2 using dockerd/moby as runtime) does not show this behavior.
Instead in this environment the `matched_text` only shows the match itself (probably flag has_long_lines get set here differently).
Notably when the same image is run within Docker on Ubuntu (using containerd as runtime) the above stated problem (huge files) also exists for this image.
We have not investigated whether this difference in behavior is due to host platform differences (Ubuntu vs. WSL2) or container runtime differences (containerd vs. dockerd) - or some other cause
Our current assumption is that the observed differences are originating somehow from libmagic (and underlying libraries).
**Whether the observed behavior is a bug or a (non) feature is unclear: At least it results in huge result files (increased here by factor 100 - typically tens of megabytes in size) and thus might result in nonfunctional issues. The matched_text data in any case is useless.**
### How To Reproduce
The following gives the commands on reproducing the observed effect based on a fresh Ubuntu 24.04 machine (including setup of Scancode).
The file `animations.mjs.map` from NPM package pkg:npm/%40angular/animations@15.2.10 is the file we scan for testing purposes.
```
sudo apt-get update && sudo apt-get upgrade
sudo add-apt-repository ppa:deadsnakes/ppa --yes
sudo apt-get install python3.8 python3.8-distutils
sudo apt install python-dev-is-python3 bzip2 xz-utils zlib1g libxml2-dev libxslt1-dev libpopt0
sudo apt-get install libgomp1
wget https://github.com/nexB/scancode-toolkit/releases/download/v32.1.0/scancode-toolkit-v32.1.0_py3.8-linux.tar.gz
tar -xvf scancode-toolkit-v32.1.0_py3.8-linux.tar.gz
cd scancode-toolkit-v32.1.0
./scancode --help
cd ..
curl -o animations.tgz https://registry.npmjs.org/@angular/animations/-/animations-15.2.10.tgz
tar xvf animations.tgz
./scancode-toolkit-v32.1.0/scancode -clipeu --license-text --json-pp scan-result.json package/fesm2015/animations.mjs.map
less scan-result.json
```
### System configuration
See infos above.
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.