aboutcode-org / aboutcode-org/scancode-toolkit

memoryerror in output plugin

未關閉
#2,028 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
bug
主要語言
Python
星號
2.6k
分支
791
平均合併
1 天 12 小時
30 天內合併 PR
5

描述

### Description

scancode fails to write to the output for scanning ~200k files.

```
ERROR: failed to run output plugin: csv:
Traceback (most recent call last):
File "c:\scancode-toolkit\src\scancode\cli.py", line 1045, in run_codebase_plugins
plugin.process_codebase(codebase, **kwargs)
File "c:\scancode-toolkit\src\formattedcode\output_csv.py", line 82, in process_codebase
write_csv(results, csv)
File "c:\scancode-toolkit\src\formattedcode\output_csv.py", line 100, in write_csv
rows = list(flatten_scan(results, headers))
File "c:\scancode-toolkit\src\formattedcode\output_csv.py", line 145, in flatten_scan
file_info['scan_errors'] = '\n'.join(errors)
File "C:\Python27\Lib\collections.py", line 78, in __setitem__
last[1] = root[0] = self.__map[key] = [last, root, key]
MemoryError

ERROR: failed to run output plugin: json-pp:
Traceback (most recent call last):
File "c:\scancode-toolkit\src\scancode\cli.py", line 1045, in run_codebase_plugins
plugin.process_codebase(codebase, **kwargs)
File "c:\scancode-toolkit\src\formattedcode\output_json.py", line 119, in process_codebase
results = get_results(codebase, as_list=False, **kwargs)
File "c:\scancode-toolkit\src\formattedcode\output_json.py", line 161, in get_results
files = OutputPlugin.get_files(codebase, **kwargs)
File "c:\scancode-toolkit\src\plugincode\output.py", line 104, in get_files
return map(serializer, resources)
MemoryError

Scanning done.
Summary: info with 4 process(es)
Errors count: 0
Scan Speed: 73.85 files/sec. 1.50 MB/sec.
Initial counts: 204473 resource(s): 187137 file(s) and 17336 directorie(s)
Final counts: 204473 resource(s): 187137 file(s) and 17336 directorie(s) for 3.72 GB
Timings:
scan_start: 2020-05-07T025407.619000
scan_end: 2020-05-07T043955.115000
inventory: 431.73s
scan: 2533.86s
output:csv: 107.15s
output:json-pp: 55.21s
output: 162.36s
total: 6570.06s
```

### How To Reproduce

The command that I used was
```
scancode -n4 -i --json-pp output.json --csv output.csv
```

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。