Further investigate changeset discrepancies in the augmented diff stream
- Dominant language
- Scala
- Stars
- 81
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
In #165 we fixed the total_edit counts for the changesets table via the ChangesetStatsForeachWriter. During testing, we found additional discrepancies between values in the `changes` db jsonb field for changesets computed via augmented diff versus imported from an OSM export. These should match, and OSM export is treated as the source of truth.
This epic is meant to encapsulate smaller tasks that are responsible for testing the augmented diff stream and the overpass diff publisher to figure out where the discrepancies arise.
The changeset 509539 in production (generated by aug diff):
```
counts | {"other_modified": 5920, "roads_modified": 35, "waterways_modified": 8, "coastlines_modified": 3}
```
vs staging (generated via OSM extract):
```
counts | {"other_modified": 5492, "roads_modified": 41, "waterways_modified": 8, "coastlines_modified": 4}
```
These are different but should be the same. At time of writing this issue, we would assume that the staging one generated via OSM extract is correct.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the ChangesetStatsForeachWriter and trace the augmented diff stream and overpass diff publisher paths mentioned in the issue. Compare changeset 509539's JSONB counts with the OSM export values, then use targeted tests or reproduction data to identify where the discrepancy arises and confirm matching results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering, databases, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100