Sharding / Planet Scale

The bad news is that currently this does not scale to the planet. Not sure what the major bottleneck or problem is here but the indexes stop growing ~ 60GB.

Solutions:
1. Shard geographically?

+ Geographic Indexes make sense; we know where the data should be
- Cannot be guaranteed of size. Will run into the same problems.
- Convoluted geographic boundaries. Even by Country: Germany / France will need to be split into multiple regions within the year

2. Shard by ID & Type?

- Will need to maintain open connections to all databases when reading (and maybe writing, if input file is not perfectly sorted)
+ Better control of size / predictability. 1 Billion Entries per Node DB, 500M entries per Way DB
+ Easy to know what DB to lookup in based on the object type and ID; simpler keys available

Not a major, major priority, but worth investigating.

/cc @lukasmartinelli

The issue names no files, tests, or entry points. Start by investigating why the indexes stop growing at roughly 60GB, then compare geographic and ID/type sharding against the repository's RocksDB design. Done would require a decided, validated sharding approach rather than only further discussion.

Written by the indexing model from the issue text.