aboutcode-org / aboutcode-org/scancode-toolkit
Python error(s) when scanning large numbers of artifacts in parallel
- Vorherrschende Sprache
- Python
- Sterne
- 2.6k
- Forks
- 791
- Ø Merge
- 1 T. 12 Std.
- Gemergte PRs (30 T.)
- 5
Beschreibung
### Description
Scanning a very large number of artifacts in parallel leads to the following Python error(s):
```
Setup plugins...
Collect file inventory...
Scan files for: licenses with 4 process(es)...
Removing temporary files...done.
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 720, in next
item = self._items.popleft()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 726, in next
item = self._items.popleft()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jenkins/venv-scancode/bin/scancode", line 8, in
sys.exit(scancode())
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/scancode/utils.py", line 70, in main
standalone_mode=standalone_mode, **extra)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/scancode/cli.py", line 530, in scancode
*args, **kwargs)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/scancode/cli.py", line 927, in run_scan
quiet=quiet, verbose=verbose, kwargs=requested_options, echo_func=echo_func,
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/scancode/cli.py", line 1118, in run_scanners
with_timing=timing, progress_manager=progress_manager)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/scancode/cli.py", line 1212, in scan_codebase
location, rid, scan_errors, scan_time, scan_result, scan_timings = next(scans)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/_termui_impl.py", line 127, in __next__
return next(iter(self))
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/_termui_impl.py", line 315, in generator
for rv in self.iter:
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/scancode/pool.py", line 48, in wrap
return func(self, timeout=timeout or 3600)
File "/usr/lib/python3.6/multiprocessing/pool.py", line 730, in next
raise TimeoutError
multiprocessing.context.TimeoutError
```
### How To Reproduce
The following command is used to execute a scan on a set of about 2500 JAR artifacts (~6GB):
```
scancode -l --timeout 21600 -n 4 --ignore '*.java' --ignore '*.scala' --ignore '*.class' \
--ignore '*.js' --ignore '*.css' --ignore '*.gif' --ignore '*.jpg' --ignore '*.svg' \
--ignore '*.R' --ignore '*.Rnw' --ignore '*.Rd' --ignore '*.Rmd' --ignore '*.sas' \
--ignore '*.afm' --ignore '*.dat' --ignore '*.dll' --ignore '*.exe' --ignore '*.mo' \
--ignore '*.pl' --ignore '*.pm' --ignore '*.res' --ignore '*.rds' --ignore '*.save' \
--ignore '*.xls' --ignore '*.xsb' --csv us-oss-licenses.csv us.zip-extract
```
There is no other magic
### System configuration
* Docker or AWS EC2 instance, 4 CPUs, 16GB Ram (even tried 32GB Ram on an EC2 instance)
* OS is always Ubuntu 18.04 with latest updates
* ScanCode Toolkit versions tried were **3.1.1** and **3.2.1rc2**
* Installation methods tried were **source download** and **pip**
### Additional info
* The exact same setups and command work fine with smaller JAR sets, e.g. less than 100 JARs and < 1GB
* No attempts were made to gauge at what number of JAR artifacts and/or data size the described behavior starts to occur as runs with larger sets take several hours to complete.
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.