GoogleCloudPlatform / GoogleCloudPlatform/gsutil

Proposal to switch from crcmod to crc32c

Open
#1,111 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

crcmod uses a simple lookup table (compiled from C).

[crc32c](https://github.com/ICRAR/crc32c) uses hardware instructions for Intel and ARMv8 CPUs, with software fallback, and is significantly faster. It also has binary builds available for most platforms, which makes it much easier to install (you could vendor it in your third_party directory even).

`time gsutil hash -c (400 mb file)`
* crcmod: 7.103 seconds
* crc32c: 3.048 seconds

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.