Minutely updates, augmented diffs, and queries
- Dominant language
- Scala
- Stars
- 81
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
Earlier this week, @lossyrob @moradology @kamicut and I got together in Philly to talk about taking OSMesa forward and kicked around some ideas. I just want to drop notes here, and invite comments from anyone who's interested.
### Recap of what we have now
1. Weekly updated full history ORC files hosted on AWS.
2. Collection of scripts that runs on Apache Spark that runs periodic analysis jobs.
### Where we want to get to
1. Minutely updates.
2. Augmented diffs for storing minutely changes.
3. Streaming minutely augmented diffs.
4. Infrastructure for arbitrary queries using tags and bboxes.
5. Infrastructure for periodic analytics jobs.
To be clear, we're not proposing above as the only future of OSMesa - rather thinking about different parts we need to build for making this work be useful for the larger OSM community. We’re using the repo to anchor this discussion for now, and will eventually fork into others when needed.
## ORC to augmented diffs
- We can use the weekly ORC files to seed history parsing much faster than parsing the planet pbf as horizontally scalable Spark jobs.
- OSMesa already has utilities that can do parts of this.
- This can be pushed down a stream for consumers.
## Nodecache for lookup
- Building full and intermediate version of geometries is important as part of the augmented diffs. This means we’ll have to store all the current (or perhaps all previous versions) of all nodes in OSM.
- There are some experiments previously using DynamoDB and RocksDB - but collectively we think DynamoDB might be better because it’s managed and hopefully won’t get too expensive.
- The datastore should be optimised for faster look up. We’ll store node versions, and membership but no geometries.
- We’ll persist the nodecache on S3 for recovery against a failure.
- The nodecache potentially only need to contain current version of all nodes.
## Minutely processing
- For every minutely change, we look up the nodecache to build the augmented diff, and write this to S3 as well into the stream.
## Analytics/Query Data Store
### Batch jobs
- We’ll build a process that listens to augmented diffs stream and incrementally stores the data for running period analytics. The store will be optimized for long running batch jobs, rather than speed of query response.
### Query server
- We probably want to think of spatio-temporal queries as a separate use case because the more predictable the queries are, the more control over indices we have especially for hbase like data stores. We discussed GeoMesa as the underlying store here with some sort of frontend server for API ergonomics.

Contributor guide
No contributing guide indexed for this repository
Research direction
No specific files, tests, or implementation entry points are named. Review the existing weekly ORC files and Apache Spark analysis scripts first, then clarify which proposed area—minutely updates, augmented diffs, nodecache, analytics, or queries—is in scope and what acceptance criteria define done.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, scala, spark
- Domain
- cloud, data-engineering, databases, distributed-systems, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100