aboutcode-org / aboutcode-org/scancode-toolkit
memoryerror in output plugin
- 主要語言
- Python
- 星號
- 2.6k
- 分支
- 791
- 平均合併
- 1 天 12 小時
- 30 天內合併 PR
- 5
描述
### Description
scancode fails to write to the output for scanning ~200k files.
```
ERROR: failed to run output plugin: csv:
Traceback (most recent call last):
File "c:\scancode-toolkit\src\scancode\cli.py", line 1045, in run_codebase_plugins
plugin.process_codebase(codebase, **kwargs)
File "c:\scancode-toolkit\src\formattedcode\output_csv.py", line 82, in process_codebase
write_csv(results, csv)
File "c:\scancode-toolkit\src\formattedcode\output_csv.py", line 100, in write_csv
rows = list(flatten_scan(results, headers))
File "c:\scancode-toolkit\src\formattedcode\output_csv.py", line 145, in flatten_scan
file_info['scan_errors'] = '\n'.join(errors)
File "C:\Python27\Lib\collections.py", line 78, in __setitem__
last[1] = root[0] = self.__map[key] = [last, root, key]
MemoryError
ERROR: failed to run output plugin: json-pp:
Traceback (most recent call last):
File "c:\scancode-toolkit\src\scancode\cli.py", line 1045, in run_codebase_plugins
plugin.process_codebase(codebase, **kwargs)
File "c:\scancode-toolkit\src\formattedcode\output_json.py", line 119, in process_codebase
results = get_results(codebase, as_list=False, **kwargs)
File "c:\scancode-toolkit\src\formattedcode\output_json.py", line 161, in get_results
files = OutputPlugin.get_files(codebase, **kwargs)
File "c:\scancode-toolkit\src\plugincode\output.py", line 104, in get_files
return map(serializer, resources)
MemoryError
Scanning done.
Summary: info with 4 process(es)
Errors count: 0
Scan Speed: 73.85 files/sec. 1.50 MB/sec.
Initial counts: 204473 resource(s): 187137 file(s) and 17336 directorie(s)
Final counts: 204473 resource(s): 187137 file(s) and 17336 directorie(s) for 3.72 GB
Timings:
scan_start: 2020-05-07T025407.619000
scan_end: 2020-05-07T043955.115000
inventory: 431.73s
scan: 2533.86s
output:csv: 107.15s
output:json-pp: 55.21s
output: 162.36s
total: 6570.06s
```
### How To Reproduce
The command that I used was
```
scancode -n4 -i --json-pp output.json --csv output.csv
```
貢獻指南
評估
這個 Issue 還沒有評估資料。