mapbox / mapbox/osm-wayback

No Deleted Objects

Open
#15 0 comments 0 reactions 0 assignees View on GitHub
bug enhancement
Dominant language
C++
Stars
2
Forks
1
PR merge metrics
No merged PRs in 30d

Description

The current Implementation doesn't allow for any deleted objects. Since the deleted objects aren't in the input file for `add_tags`, they don't ever get queried from the DB.

Further, the `add_tags` taking streaming input of geojson works really well for augmenting existing files, but requiring all of the steps for new data: `.osh` -> `.osm` -> `.geojson` -> `history.geojson` is a bit cumbersome (and we lose deleted objects).

Solutions
---------
1. Add a special key for deleted tags (Can you query rocksdb and get back an iterator matching a key regex?) --> Or maybe keep a running list of deleted IDs to store in their own key that will be fetched and then written out IF a `deleted` flag is set? (Deleted Objects would also have to contain their last-known-geometry; we'd have to start storing (all) node geometries as well, so that these could be recreated); unfortunately we don't know what version is the most recent, so we'd have to simply start from v0 and increment until it failed).

What if we stored the node table in memory?

|attr|type|size|
|----|----|---|
| id | `` | 8 bytes |
| version | `` | 1 byte |
| changeset | `` | 4 bytes |
| timestamp | `` | 4 bytes |
| lon | `` | 8 bytes |
| lat | `< double >` | 8 bytes |
| | | 33 bytes |

33 * 10 Billion = 330 Billion Bytes = 330 gigabytes for the planet, currently.

> This is not _that_ crazy for parsing the planet (wouldn't have to do this very often) A 512G machine could handle this for the foreseeable 2 - 3 years if crunching the entire history on a regular basis.

2. Can we simply iterate over all keys in rocksdb? This takes steps out of the process BUT then we'd have to store geometries -- which brings us back to square one; but I do think this is a worthy path to travel down: A separate `node!version` -> `changeset,time,lat,lon` store for non deleted nodes.

3. This could all be dramatically simplified if the `VISIBLE` flag was `FALSE` for any historical object versions that were no longer the _visible_ version of the object :)

/cc @lukasmartinelli @batpad ...just my evening :bulb:

Contributor guide

Open the contributing guide

Research direction

Read the add_tags streaming path and the RocksDB key layout first, then compare how deleted objects are represented in the proposed approaches. Done means a chosen design preserves deleted objects and produces the needed history output without requiring the full .osh -> .osm -> .geojson -> history.geojson conversion.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.