Climate-REF / Climate-REF/climate-ref

executions fail-running reports success without committing

Open
#849 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
26
Forks
14
Avg merge
1d 10h
Merged PRs (30d)
36

Description

`ref executions fail-running` prints `Successfully marked N execution(s) as failed and flagged their execution groups as dirty` but leaves the database unchanged when the session already has a transaction open.

The write is wrapped in:

```python
with session.begin_nested() if session.in_transaction() else session.begin():
```

`begin_nested()` opens a SAVEPOINT. Leaving the block releases the savepoint, it does not commit the enclosing transaction, so the changes are discarded when the session closes. The success message is printed unconditionally after the block, so there is no signal that nothing happened.

Reproduced against v0.16.2 on Postgres. Counting the affected rows before and after, twice in a row:

```
3336 # executions with successful IS NULL for this provider
Successfully marked 3336 execution(s) as failed and flagged their execution groups as dirty.
3336 # unchanged
```

This matters because `fail-running` is the documented escape hatch for exactly this situation. `ExecutionGroup.should_run` returns False while the last execution has `successful IS NULL`, so an execution abandoned by an OOM-killed worker blocks its group from ever being resolved again. `fail_stale_in_progress_executions` only reaps executions older than `stale_after_seconds` (6 hours by default), so anything more recent has no working recovery path and the operator is told the recovery succeeded.

A plain `session.commit()` after the loop, or committing the outer transaction when one was already open, would fix it.

Contributor guide

Open the contributing guide

Research direction

Start at the `ref executions fail-running` entry point and inspect the session transaction block described in the issue. Reproduce the command with an already-open transaction, then verify that affected execution rows and execution-group flags remain changed after the session closes; the success output should accurately reflect the committed result.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgres, python
Domain
cli, database
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.