Change DEFLATE from zlib to libdeflate
- Dominant language
- C++
- Stars
- 3.1k
- Forks
- 430
- PR merge metrics
- No merged PRs in 30d
Description
zlib is no longer a good choice for a fast and performant DEFLATE algorithm.
Just having picked one other that I found, https://github.com/ebiggers/libdeflate, I found it produces ~20% smaller files in less time.
For example, on a large tile of GeoJSON points:
2116919B Uncompressed
375282B libdeflate -12
378332B libdeflate -9
449989B tippecanoe/zlib -9
For a small tile with a few polygons, the difference was less dramatic but still positive:
53477B Uncompressed
40501B libdeflate -12
40552B libdeflate -9
41121B tippecanoe/zlib -9
I see another mapbox project is in the process of migrating the same library - https://github.com/mapbox/gzip-hpp/pull/25
I have not compared to any other zlib libraries.
Contributor guide
Research direction
Start by locating tippecanoe's current zlib-based DEFLATE integration and its compression tests or build dependencies. Review libdeflate and the benchmark figures in this issue, then compare the existing and replacement paths on representative GeoJSON tiles. Done means the project uses libdeflate reliably and its tests and builds still pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100