aboutcode-org / aboutcode-org/scancode-toolkit

Question: dealing with timeouts during the scan

Abierto
#3,143 4 comentarios 3 reacciones 0 asignados Ver en GitHub
bug performance
Lenguaje dominante
Python
Estrellas
2.6k
Forks
791
Merge medio
1 d 12 h
PR fusionados (30 d)
5

Descripción

I run Scancode within ORT on a big repository (+ 3.5 GB) containing .json test data (a lot a big files).

Scan crashes with a timeout:
```
Running '/opt/python/shims/scancode --copyright --license --info --strip-root --timeout 300 --max-in-memory 5000 --processes 7 /tmp/ort-DefaultProvenanceDownloader6273558335820335397 --json-pp /tmp/ort-ScanCode9471182827864591694/result.json' in '/home/ort'...

/opt/python/versions/3.10.6/lib/python3.10/site-packages/cluecode/copyrights.py:3382: FutureWarning: Possible set difference at position 3
remove_tags = re.compile(
Setup plugins...
Collect file inventory...
Scan files for: info, licenses, copyrights with 7 process(es)...
Removing temporary files...done.
Traceback (most recent call last):
File "/opt/python/versions/3.10.6/bin/scancode", line 8, in
sys.exit(scancode())
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/commoncode/cliutils.py", line 69, in main
return click.Command.main(
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 451, in scancode
success, _results = run_scan(
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 887, in run_scan
scan_success = run_scanners(
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 1125, in run_scanners
scan_success = scan_codebase(
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/cli.py", line 1236, in scan_codebase
scan_timings) = next(scans)
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/click/_termui_impl.py", line 116, in __next__
return next(iter(self))
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/commoncode/cliutils.py", line 170, in generator
yield from self.iter
File "/opt/python/versions/3.10.6/lib/python3.10/site-packages/scancode/pool.py", line 52, in wrap
return func(self, timeout=timeout or 3600)
File "/opt/python/versions/3.10.6/lib/python3.10/multiprocessing/pool.py", line 868, in next
raise TimeoutError from None
multiprocessing.context.TimeoutError
```
Is this occurring because of the `--timeout 300` parameter I specified ?
I know the clean solution would be to use `--ignore` parameter but is there another way to alleviate this error ?

Also the memory usage goes through the roof so I tried to see if ` --max-in-memory -1` helps to prevent scancode putting the big files in memory by no luck here.

Any help is welcome.

OS: eclipse-temurin:11-jdk-jammy
Scancode: 30.1.0.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.