aboutcode-org / aboutcode-org/scancode-toolkit

Question: dealing with timeouts during the scan

未关闭
#3,143 4 条评论 3 个 reaction 已指派 0 人 在 GitHub 查看
bug performance
主要语言
Python
星标
2.6k
派生
791
平均合并
1 天 12 小时
30 天内合并 PR
5

描述

I run Scancode within ORT on a big repository (+ 3.5 GB) containing .json test data (a lot a big files).

Scan crashes with a timeout:
```
Running '/opt/python/shims/scancode --copyright --license --info --strip-root --timeout 300 --max-in-memory 5000 --processes 7 /tmp/ort-DefaultProvenanceDownloader6273558335820335397 --json-pp /tmp/ort-ScanCode9471182827864591694/result.json' in '/home/ort'...

/opt/python/versions/3.10.6/lib/python3.10/site-packages/cluecode/copyrights.py:3382: FutureWarning: Possible set difference at position 3
remove_tags = re.compile(
Setup plugins...
Collect file inventory...
Scan files for: info, licenses, copyrights with 7 process(es)...
Removing temporary files...done.
Traceback (most recent call last):
File "/opt/python/versions/3.10.6/bin/scancode", line 8, in
sys.exit(scancode())
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/commoncode/cliutils.py", line 69, in main
return click.Command.main(
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 451, in scancode
success, _results = run_scan(
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 887, in run_scan
scan_success = run_scanners(
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 1125, in run_scanners
scan_success = scan_codebase(
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 1236, in scan_codebase
scan_timings) = next(scans)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/_termui_impl.py", line 116, in __next__
return next(iter(self))
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/commoncode/cliutils.py", line 170, in generator
yield from self.iter
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/pool.py", line 52, in wrap
return func(self, timeout=timeout or 3600)
File "/opt/python/versions/3.10.6/lib/python3.10/multiprocessing/pool.py", line 868, in next
raise TimeoutError from None
multiprocessing.context.TimeoutError
```
Is this occurring because of the `--timeout 300` parameter I specified ?
I know the clean solution would be to use `--ignore` parameter but is there another way to alleviate this error ?

Also the memory usage goes through the roof so I tried to see if ` --max-in-memory -1` helps to prevent scancode putting the big files in memory by no luck here.

Any help is welcome.

OS: eclipse-temurin:11-jdk-jammy
Scancode: 30.1.0.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。