aboutcode-org / aboutcode-org/scancode-toolkit

memoryerror in output plugin

Abierto
#2,028 0 comentarios 0 reacciones 0 asignados Ver en GitHub
bug
Lenguaje dominante
Python
Estrellas
2.6k
Forks
791
Merge medio
1 d 12 h
PR fusionados (30 d)
5

Descripción

### Description

scancode fails to write to the output for scanning ~200k files.

```
ERROR: failed to run output plugin: csv:
Traceback (most recent call last):
File "c:\scancode-toolkit\src\scancode\cli.py", line 1045, in run_codebase_plugins
plugin.process_codebase(codebase, **kwargs)
File "c:\scancode-toolkit\src\formattedcode\output_csv.py", line 82, in process_codebase
write_csv(results, csv)
File "c:\scancode-toolkit\src\formattedcode\output_csv.py", line 100, in write_csv
rows = list(flatten_scan(results, headers))
File "c:\scancode-toolkit\src\formattedcode\output_csv.py", line 145, in flatten_scan
file_info['scan_errors'] = '\n'.join(errors)
File "C:\Python27\Lib\collections.py", line 78, in __setitem__
last[1] = root[0] = self.__map[key] = [last, root, key]
MemoryError

ERROR: failed to run output plugin: json-pp:
Traceback (most recent call last):
File "c:\scancode-toolkit\src\scancode\cli.py", line 1045, in run_codebase_plugins
plugin.process_codebase(codebase, **kwargs)
File "c:\scancode-toolkit\src\formattedcode\output_json.py", line 119, in process_codebase
results = get_results(codebase, as_list=False, **kwargs)
File "c:\scancode-toolkit\src\formattedcode\output_json.py", line 161, in get_results
files = OutputPlugin.get_files(codebase, **kwargs)
File "c:\scancode-toolkit\src\plugincode\output.py", line 104, in get_files
return map(serializer, resources)
MemoryError

Scanning done.
Summary: info with 4 process(es)
Errors count: 0
Scan Speed: 73.85 files/sec. 1.50 MB/sec.
Initial counts: 204473 resource(s): 187137 file(s) and 17336 directorie(s)
Final counts: 204473 resource(s): 187137 file(s) and 17336 directorie(s) for 3.72 GB
Timings:
scan_start: 2020-05-07T025407.619000
scan_end: 2020-05-07T043955.115000
inventory: 431.73s
scan: 2533.86s
output:csv: 107.15s
output:json-pp: 55.21s
output: 162.36s
total: 6570.06s
```

### How To Reproduce

The command that I used was
```
scancode -n4 -i --json-pp output.json --csv output.csv
```

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.