aboutcode-org / aboutcode-org/scancode-toolkit

Question: dealing with timeouts during the scan

Đang mở
#3,143 4 bình luận 3 reaction 0 người được giao Xem trên GitHub
bug performance
Ngôn ngữ chính
Python
Star
2.6k
Fork
791
Merge trung bình
1 ngày 12 giờ
Pull request đã merge (30 ngày)
5

Mô tả

I run Scancode within ORT on a big repository (+ 3.5 GB) containing .json test data (a lot a big files).

Scan crashes with a timeout:
```
Running '/opt/python/shims/scancode --copyright --license --info --strip-root --timeout 300 --max-in-memory 5000 --processes 7 /tmp/ort-DefaultProvenanceDownloader6273558335820335397 --json-pp /tmp/ort-ScanCode9471182827864591694/result.json' in '/home/ort'...

/opt/python/versions/3.10.6/lib/python3.10/site-packages/cluecode/copyrights.py:3382: FutureWarning: Possible set difference at position 3
remove_tags = re.compile(
Setup plugins...
Collect file inventory...
Scan files for: info, licenses, copyrights with 7 process(es)...
Removing temporary files...done.
Traceback (most recent call last):
File "/opt/python/versions/3.10.6/bin/scancode", line 8, in
sys.exit(scancode())
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/commoncode/cliutils.py", line 69, in main
return click.Command.main(
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 451, in scancode
success, _results = run_scan(
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 887, in run_scan
scan_success = run_scanners(
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 1125, in run_scanners
scan_success = scan_codebase(
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 1236, in scan_codebase
scan_timings) = next(scans)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/_termui_impl.py", line 116, in __next__
return next(iter(self))
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/commoncode/cliutils.py", line 170, in generator
yield from self.iter
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/pool.py", line 52, in wrap
return func(self, timeout=timeout or 3600)
File "/opt/python/versions/3.10.6/lib/python3.10/multiprocessing/pool.py", line 868, in next
raise TimeoutError from None
multiprocessing.context.TimeoutError
```
Is this occurring because of the `--timeout 300` parameter I specified ?
I know the clean solution would be to use `--ignore` parameter but is there another way to alleviate this error ?

Also the memory usage goes through the roof so I tried to see if ` --max-in-memory -1` helps to prevent scancode putting the big files in memory by no luck here.

Any help is welcome.

OS: eclipse-temurin:11-jdk-jammy
Scancode: 30.1.0.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.