aboutcode-org / aboutcode-org/scancode-toolkit

memoryerror in output plugin

Đang mở
#2,028 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
bug
Ngôn ngữ chính
Python
Star
2.6k
Fork
791
Merge trung bình
1 ngày 12 giờ
Pull request đã merge (30 ngày)
5

Mô tả

### Description

scancode fails to write to the output for scanning ~200k files.

```
ERROR: failed to run output plugin: csv:
Traceback (most recent call last):
File "c:\scancode-toolkit\src\scancode\cli.py", line 1045, in run_codebase_plugins
plugin.process_codebase(codebase, **kwargs)
File "c:\scancode-toolkit\src\formattedcode\output_csv.py", line 82, in process_codebase
write_csv(results, csv)
File "c:\scancode-toolkit\src\formattedcode\output_csv.py", line 100, in write_csv
rows = list(flatten_scan(results, headers))
File "c:\scancode-toolkit\src\formattedcode\output_csv.py", line 145, in flatten_scan
file_info['scan_errors'] = '\n'.join(errors)
File "C:\Python27\Lib\collections.py", line 78, in __setitem__
last[1] = root[0] = self.__map[key] = [last, root, key]
MemoryError

ERROR: failed to run output plugin: json-pp:
Traceback (most recent call last):
File "c:\scancode-toolkit\src\scancode\cli.py", line 1045, in run_codebase_plugins
plugin.process_codebase(codebase, **kwargs)
File "c:\scancode-toolkit\src\formattedcode\output_json.py", line 119, in process_codebase
results = get_results(codebase, as_list=False, **kwargs)
File "c:\scancode-toolkit\src\formattedcode\output_json.py", line 161, in get_results
files = OutputPlugin.get_files(codebase, **kwargs)
File "c:\scancode-toolkit\src\plugincode\output.py", line 104, in get_files
return map(serializer, resources)
MemoryError

Scanning done.
Summary: info with 4 process(es)
Errors count: 0
Scan Speed: 73.85 files/sec. 1.50 MB/sec.
Initial counts: 204473 resource(s): 187137 file(s) and 17336 directorie(s)
Final counts: 204473 resource(s): 187137 file(s) and 17336 directorie(s) for 3.72 GB
Timings:
scan_start: 2020-05-07T025407.619000
scan_end: 2020-05-07T043955.115000
inventory: 431.73s
scan: 2533.86s
output:csv: 107.15s
output:json-pp: 55.21s
output: 162.36s
total: 6570.06s
```

### How To Reproduce

The command that I used was
```
scancode -n4 -i --json-pp output.json --csv output.csv
```

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.