aboutcode-org / aboutcode-org/scancode-toolkit

Question: dealing with timeouts during the scan

オープン
#3,143 コメント 4 件 リアクション 3 件 担当者 0 名 GitHub で見る
bug performance
主要言語
Python
スター
2.6k
フォーク
791
平均マージ
1日 12時間
マージ済み PR(30日)
5

説明

I run Scancode within ORT on a big repository (+ 3.5 GB) containing .json test data (a lot a big files).

Scan crashes with a timeout:
```
Running '/opt/python/shims/scancode --copyright --license --info --strip-root --timeout 300 --max-in-memory 5000 --processes 7 /tmp/ort-DefaultProvenanceDownloader6273558335820335397 --json-pp /tmp/ort-ScanCode9471182827864591694/result.json' in '/home/ort'...

/opt/python/versions/3.10.6/lib/python3.10/site-packages/cluecode/copyrights.py:3382: FutureWarning: Possible set difference at position 3
remove_tags = re.compile(
Setup plugins...
Collect file inventory...
Scan files for: info, licenses, copyrights with 7 process(es)...
Removing temporary files...done.
Traceback (most recent call last):
File "/opt/python/versions/3.10.6/bin/scancode", line 8, in
sys.exit(scancode())
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/commoncode/cliutils.py", line 69, in main
return click.Command.main(
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 451, in scancode
success, _results = run_scan(
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 887, in run_scan
scan_success = run_scanners(
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 1125, in run_scanners
scan_success = scan_codebase(
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 1236, in scan_codebase
scan_timings) = next(scans)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/_termui_impl.py", line 116, in __next__
return next(iter(self))
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/commoncode/cliutils.py", line 170, in generator
yield from self.iter
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/pool.py", line 52, in wrap
return func(self, timeout=timeout or 3600)
File "/opt/python/versions/3.10.6/lib/python3.10/multiprocessing/pool.py", line 868, in next
raise TimeoutError from None
multiprocessing.context.TimeoutError
```
Is this occurring because of the `--timeout 300` parameter I specified ?
I know the clean solution would be to use `--ignore` parameter but is there another way to alleviate this error ?

Also the memory usage goes through the roof so I tried to see if ` --max-in-memory -1` helps to prevent scancode putting the big files in memory by no luck here.

Any help is welcome.

OS: eclipse-temurin:11-jdk-jammy
Scancode: 30.1.0.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。