aboutcode-org / aboutcode-org/scancode-toolkit

Python error(s) when scanning large numbers of artifacts in parallel

未關閉
#2,293 7 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
bug
主要語言
Python
星號
2.6k
分支
791
平均合併
1 天 12 小時
30 天內合併 PR
5

描述

### Description

Scanning a very large number of artifacts in parallel leads to the following Python error(s):
```
Setup plugins...
Collect file inventory...
Scan files for: licenses with 4 process(es)...
Removing temporary files...done.
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 720, in next
item = self._items.popleft()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 726, in next
item = self._items.popleft()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/jenkins/venv-scancode/bin/scancode", line 8, in
sys.exit(scancode())
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/scancode/utils.py", line 70, in main
standalone_mode=standalone_mode, **extra)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/scancode/cli.py", line 530, in scancode
*args, **kwargs)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/scancode/cli.py", line 927, in run_scan
quiet=quiet, verbose=verbose, kwargs=requested_options, echo_func=echo_func,
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/scancode/cli.py", line 1118, in run_scanners
with_timing=timing, progress_manager=progress_manager)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/scancode/cli.py", line 1212, in scan_codebase
location, rid, scan_errors, scan_time, scan_result, scan_timings = next(scans)
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/_termui_impl.py", line 127, in __next__
return next(iter(self))
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/click/_termui_impl.py", line 315, in generator
for rv in self.iter:
File "/home/jenkins/venv-scancode/lib/python3.6/site-packages/scancode/pool.py", line 48, in wrap
return func(self, timeout=timeout or 3600)
File "/usr/lib/python3.6/multiprocessing/pool.py", line 730, in next
raise TimeoutError
multiprocessing.context.TimeoutError
```

### How To Reproduce

The following command is used to execute a scan on a set of about 2500 JAR artifacts (~6GB):
```
scancode -l --timeout 21600 -n 4 --ignore '*.java' --ignore '*.scala' --ignore '*.class' \
--ignore '*.js' --ignore '*.css' --ignore '*.gif' --ignore '*.jpg' --ignore '*.svg' \
--ignore '*.R' --ignore '*.Rnw' --ignore '*.Rd' --ignore '*.Rmd' --ignore '*.sas' \
--ignore '*.afm' --ignore '*.dat' --ignore '*.dll' --ignore '*.exe' --ignore '*.mo' \
--ignore '*.pl' --ignore '*.pm' --ignore '*.res' --ignore '*.rds' --ignore '*.save' \
--ignore '*.xls' --ignore '*.xsb' --csv us-oss-licenses.csv us.zip-extract
```

There is no other magic

### System configuration

* Docker or AWS EC2 instance, 4 CPUs, 16GB Ram (even tried 32GB Ram on an EC2 instance)
* OS is always Ubuntu 18.04 with latest updates
* ScanCode Toolkit versions tried were **3.1.1** and **3.2.1rc2**
* Installation methods tried were **source download** and **pip**

### Additional info
* The exact same setups and command work fine with smaller JAR sets, e.g. less than 100 JARs and < 1GB
* No attempts were made to gauge at what number of JAR artifacts and/or data size the described behavior starts to occur as runs with larger sets take several hours to complete.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。