azavea / azavea/osmesa

Tool to apply diffs to an ORC file to update it

Open
#48 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
81
Forks
26
PR merge metrics
No merged PRs in 30d

Description

As discussed in #25, we should create a tool that checks the replication sequence number from the ORC file (which is a TBD; it should be in the OSM PBF metadata but isn't currently copied to the ORC user metadata by `osm2orc`), fetches all OsmChange diffs (ideally minutely to capture incremental changes) from `planet.osm.org` (or an S3 mirror, also TBD), and applies them to the ORC file as additional rows, writing the result back out to S3 (probably partitioned, probably unsorted since that makes the task faster and the output is not intended to be downloaded / used for purposes that assume sorting).

Contributor guide

No contributing guide indexed for this repository

Research direction

Review issue #25 and the existing osm2orc entry point first, focusing on how ORC user metadata could carry the replication sequence number. Trace the OsmChange source at planet.osm.org or an S3 mirror and define how diffs would be applied and written back to S3. Done means the sequence can be identified, incremental diffs are incorporated, and the resulting ORC output format is specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.