azavea / azavea/osmesa

Large augmented diff causes infinite loop

Open
#217 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
81
Forks
26
PR merge metrics
No merged PRs in 30d

Description

We observed the following log behavior:
```
21/04/29 21:07:19 DEBUG sources.AugmentedDiffSource: Fetching sequence 4529538
21/04/29 21:07:21 ERROR sources.AugmentedDiffSource: sequence 4529538 caused an error
java.lang.OutOfMemoryError: Java heap space
```
The augmented diff was larger than usual, but the problem is that the failure mode was to retry the download infinitely. We need to allow this failure to percolate up and kill the application so that other alerts might be triggered.

Contributor guide

No contributing guide indexed for this repository

Research direction

The log identifies sources.AugmentedDiffSource and sequence 4529538; start by tracing its error and retry handling for oversized augmented diffs. Verify that this failure propagates instead of retrying indefinitely, allowing the application to terminate so external alerts can trigger.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.