Write tiles as Tapalcatl 2 archives: zipped pyramids of vector tiles on S3
- Dominant language
- Scala
- Stars
- 81
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
(_I'm paraphrasing a bit here, because I may be interpreting tapalcatl's intent slightly differently._) [Tapalcatl](https://github.com/tilezen/tapalcatl) ([Python](https://github.com/tilezen/tapalcatl-py)) facilitate storage of multiple formats + tiles within a single meta-tile. The meta-tile is a zip file. (This is effectively the generalized tile equivalent of a Cloud Optimized GeoTIFF.)
I propose grouping pyramids of generated vector tiles (initially 8 zoom levels, but that may be too many depending on the size of the zip file index and file itself) and zipping them together prior to writing to S3. This will dramatically reduce the number of objects written to S3 (reducing latency and cost) while facilitating improved caching of data.
Under this proposal, if `/4/5/6.mvt` (the target tile) were requested, `/0/0/0.zip` (the meta tile) would be fetched and `/4/5/6.mvt` extracted from it.
Zip files include a directory w/ entry offsets, so it's possible to do partial reads of the zip file (3 requests: 2 for the directory (which can be cached), one for the entry itself (which can be cached as part of larger block reads, e.g. 10MB aligned)) (see https://github.com/mojodna/tilelive-tapalcatl for a _really_ preliminary implementation of this).
Reader support for tapalcatl meta tiles could be implemented in client code, via a proxy server (`tapalcatl-py`) that allows individual tile requests, or as a Service Worker that can intercept tile requests + cache blocks.
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no OSMesa files, tests, or entry points. Start by tracing how generated vector tiles are written to S3, then read Tapalcatl, tapalcatl-py, and the preliminary tilelive-tapalcatl implementation. Done means generated tiles are grouped into readable zip pyramids and the storage approach is validated against the proposed partial-read and caching model.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, scala, spark
- Domain
- backend, cloud, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100