Climate-REF / Climate-REF/climate-ref
executions fail-running reports success without committing
- Dominant language
- Python
- Stars
- 26
- Forks
- 14
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 36
Description
`ref executions fail-running` prints `Successfully marked N execution(s) as failed and flagged their execution groups as dirty` but leaves the database unchanged when the session already has a transaction open.
The write is wrapped in:
```python
with session.begin_nested() if session.in_transaction() else session.begin():
```
`begin_nested()` opens a SAVEPOINT. Leaving the block releases the savepoint, it does not commit the enclosing transaction, so the changes are discarded when the session closes. The success message is printed unconditionally after the block, so there is no signal that nothing happened.
Reproduced against v0.16.2 on Postgres. Counting the affected rows before and after, twice in a row:
```
3336 # executions with successful IS NULL for this provider
Successfully marked 3336 execution(s) as failed and flagged their execution groups as dirty.
3336 # unchanged
```
This matters because `fail-running` is the documented escape hatch for exactly this situation. `ExecutionGroup.should_run` returns False while the last execution has `successful IS NULL`, so an execution abandoned by an OOM-killed worker blocks its group from ever being resolved again. `fail_stale_in_progress_executions` only reaps executions older than `stale_after_seconds` (6 hours by default), so anything more recent has no working recovery path and the operator is told the recovery succeeded.
A plain `session.commit()` after the loop, or committing the outer transaction when one was already open, would fix it.
Contributor guide
Research direction
Start at the `ref executions fail-running` entry point and inspect the session transaction block described in the issue. Reproduce the command with an already-open transaction, then verify that affected execution rows and execution-group flags remain changed after the session closes; the success output should accurately reflect the committed result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgres, python
- Domain
- cli, database
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100