aboutcode-org / aboutcode-org/scancode-toolkit

Python error(s) when scanning large numbers of artifacts in parallel

Abierto
#2,293 7 comentarios 0 reacciones 0 asignados Ver en GitHub
bug
Lenguaje dominante
Python
Estrellas
2.6k
Forks
791
Merge medio
1 d 12 h
PR fusionados (30 d)
5

Descripción

### Description

Scanning a very large number of artifacts in parallel leads to the following Python error(s):
```
Setup plugins...
Collect file inventory...
Scan files for: licenses with 4 process(es)...
Removing temporary files...done.
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 720, in next
item = self._items.popleft()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 726, in next
item = self._items.popleft()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/jenkins/venv-scancode/bin/scancode", line 8, in
sys.exit(scancode())
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/scancode/utils.py", line 70, in main
standalone_mode=standalone_mode, **extra)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/scancode/cli.py", line 530, in scancode
*args, **kwargs)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/scancode/cli.py", line 927, in run_scan
quiet=quiet, verbose=verbose, kwargs=requested_options, echo_func=echo_func,
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/scancode/cli.py", line 1118, in run_scanners
with_timing=timing, progress_manager=progress_manager)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/scancode/cli.py", line 1212, in scan_codebase
location, rid, scan_errors, scan_time, scan_result, scan_timings = next(scans)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/_termui_impl.py", line 127, in __next__
return next(iter(self))
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/_termui_impl.py", line 315, in generator
for rv in self.iter:
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/scancode/pool.py", line 48, in wrap
return func(self, timeout=timeout or 3600)
File "/usr/lib/python3.6/multiprocessing/pool.py", line 730, in next
raise TimeoutError
multiprocessing.context.TimeoutError
```

### How To Reproduce

The following command is used to execute a scan on a set of about 2500 JAR artifacts (~6GB):
```
scancode -l --timeout 21600 -n 4 --ignore '*.java' --ignore '*.scala' --ignore '*.class' \
--ignore '*.js' --ignore '*.css' --ignore '*.gif' --ignore '*.jpg' --ignore '*.svg' \
--ignore '*.R' --ignore '*.Rnw' --ignore '*.Rd' --ignore '*.Rmd' --ignore '*.sas' \
--ignore '*.afm' --ignore '*.dat' --ignore '*.dll' --ignore '*.exe' --ignore '*.mo' \
--ignore '*.pl' --ignore '*.pm' --ignore '*.res' --ignore '*.rds' --ignore '*.save' \
--ignore '*.xls' --ignore '*.xsb' --csv us-oss-licenses.csv us.zip-extract
```

There is no other magic

### System configuration

* Docker or AWS EC2 instance, 4 CPUs, 16GB Ram (even tried 32GB Ram on an EC2 instance)
* OS is always Ubuntu 18.04 with latest updates
* ScanCode Toolkit versions tried were **3.1.1** and **3.2.1rc2**
* Installation methods tried were **source download** and **pip**

### Additional info
* The exact same setups and command work fine with smaller JAR sets, e.g. less than 100 JARs and < 1GB
* No attempts were made to gauge at what number of JAR artifacts and/or data size the described behavior starts to occur as runs with larger sets take several hours to complete.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.