azavea / azavea/osmesa

Further investigate changeset discrepancies in the augmented diff stream

Open
#186 4 comments 0 reactions 0 assignees View on GitHub
Epic
Dominant language
Scala
Stars
81
Forks
26
PR merge metrics
No merged PRs in 30d

Description

In #165 we fixed the total_edit counts for the changesets table via the ChangesetStatsForeachWriter. During testing, we found additional discrepancies between values in the `changes` db jsonb field for changesets computed via augmented diff versus imported from an OSM export. These should match, and OSM export is treated as the source of truth.

This epic is meant to encapsulate smaller tasks that are responsible for testing the augmented diff stream and the overpass diff publisher to figure out where the discrepancies arise.

The changeset 509539 in production (generated by aug diff):
```
counts | {"other_modified": 5920, "roads_modified": 35, "waterways_modified": 8, "coastlines_modified": 3}
```
vs staging (generated via OSM extract):
```
counts | {"other_modified": 5492, "roads_modified": 41, "waterways_modified": 8, "coastlines_modified": 4}
```
These are different but should be the same. At time of writing this issue, we would assume that the staging one generated via OSM extract is correct.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the ChangesetStatsForeachWriter and trace the augmented diff stream and overpass diff publisher paths mentioned in the issue. Compare changeset 509539's JSONB counts with the OSM export values, then use targeted tests or reproduction data to identify where the discrepancy arises and confirm matching results.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, databases, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.