mapbox / mapbox/tippecanoe

Change DEFLATE from zlib to libdeflate

Open
#674 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
3.1k
Forks
430
PR merge metrics
No merged PRs in 30d

Description

zlib is no longer a good choice for a fast and performant DEFLATE algorithm.
Just having picked one other that I found, https://github.com/ebiggers/libdeflate, I found it produces ~20% smaller files in less time.

For example, on a large tile of GeoJSON points:
2116919B Uncompressed
375282B libdeflate -12
378332B libdeflate -9
449989B tippecanoe/zlib -9

For a small tile with a few polygons, the difference was less dramatic but still positive:
53477B Uncompressed
40501B libdeflate -12
40552B libdeflate -9
41121B tippecanoe/zlib -9

I see another mapbox project is in the process of migrating the same library - https://github.com/mapbox/gzip-hpp/pull/25

I have not compared to any other zlib libraries.

Contributor guide

Open the contributing guide

Research direction

Start by locating tippecanoe's current zlib-based DEFLATE integration and its compression tests or build dependencies. Review libdeflate and the benchmark figures in this issue, then compare the existing and replacement paths on representative GeoJSON tiles. Done means the project uses libdeflate reliably and its tests and builds still pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.