Tool to apply diffs to an ORC file to update it
- Dominant language
- Scala
- Stars
- 81
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
As discussed in #25, we should create a tool that checks the replication sequence number from the ORC file (which is a TBD; it should be in the OSM PBF metadata but isn't currently copied to the ORC user metadata by `osm2orc`), fetches all OsmChange diffs (ideally minutely to capture incremental changes) from `planet.osm.org` (or an S3 mirror, also TBD), and applies them to the ORC file as additional rows, writing the result back out to S3 (probably partitioned, probably unsorted since that makes the task faster and the output is not intended to be downloaded / used for purposes that assume sorting).
Contributor guide
No contributing guide indexed for this repository
Research direction
Review issue #25 and the existing osm2orc entry point first, focusing on how ORC user metadata could carry the replication sequence number. Trace the OsmChange source at planet.osm.org or an S3 mirror and define how diffs would be applied and written back to S3. Done means the sequence can be identified, incremental diffs are incorporated, and the resulting ORC output format is specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100