azavea / azavea/osmesa

Analytic VectorTiles from OSM History

Open
#28 10 comments 1 reaction 0 assignees View on GitHub
Dominant language
Scala
Stars
81
Forks
26
PR merge metrics
No merged PRs in 30d

Description

Generate a layer of Analytic VectorTiles with OSM geometries from OSM history in order to produce them with accurate create, modified timestamp metadata.

### Generating Features with Metadata:
In OSM data model only Nodes contain spatial data and they may be referred to by Ways to define, lines, boundaries and polygons. Relations may refer to any of the Nodes, Ways or other Relations. Depending on tags information some Relations have geographic meaning (ex: defining multi polygon with its holes) and others may have only semantic/labelling meaning.

When updates to OSM happen new records with updated changeset/version information are introduced but they retain previous ID and other records referring to changed record are not updated.

![osm connected component](https://user-images.githubusercontent.com/1158084/36169819-5cf29190-10cb-11e8-9161-6f0386f14194.png)

For instance it is common to change location of one Node in a Way, which is part of Relation representing multi polygon without updating either the Way or the Relation. This change of constituent point needs to be propagated as a change to the geographic feature.

### Proposed Approach
The proposed approach to dealing with this problem is to separate the task of grouping all related OSM records (records are related if one references another by its ID field) and the task of converting the the grouped records into geometric feature with some metadata.

Base assumption is that each group will from a directionally connected component that can fit in memory.
The second stage is converting this connected component to a geographic feature with some metadata.
Because the rules of generating features and then their metadata are varied, and ultimately application dependent keeping them separate from the shuffle logic has organizational benefits.

Here are some true things these connected components
- Multiple records sharing IDs may have multiple versions
- Multiple records sharing IDs may have different visibility status
- We will have many points, fewer ways, fewer relations

Note: If the in-memory assumption does not hold we will have to define condition on how or when to duplicate records when they are shared by more than one graph.

### Notes
Current version of `vectorpipe` employs a similar strategy in its `osm` module but runs into performance problems. It appears that performing the joins in DataFrame, rather than RDD, context addresses these concerns likely because it removes the need for intermediate, per-row, object allocation when performing the joins but rather maintains an `Array` of primitive values.

As part of this task we should pursued the DataFrame approach and add resulting work to `vector pipe` package.

### Sub tasks:
- [ ] [Generate sample VectorTile output](https://github.com/azavea/osmesa/issues/29)
- [ ] Group related OSM records using DataFrames
- Generalize Seths code
- [ ] Define data structure to navigate the connected component
- [ ] Mapping from connected component to `Feature[G, VersionMetadata]`
- [ ] Test on OSM History

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the existing vectorpipe `osm` module and the related sample VectorTile output issue (#29). Compare its join strategy with the proposed DataFrame approach, then trace how OSM History records are represented. Done means related records can be grouped, converted into features with version metadata, and tested against OSM History.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.