GaloyMoney / GaloyMoney/drua

Feature request: safely clear staging Terraform/OpenTofu locks via Drua

Open
#369 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
10
Forks
4
Avg merge
1d 58m
Merged PRs (30d)
7

Description

## Problem

When a staging/testflight Terraform/OpenTofu job is aborted or interrupted, it can leave a stale remote state lock behind. Today, clearing that lock requires giving the Codex/operator shell direct `gcloud` access to the relevant GCS state bucket or intercepting a Concourse task container that has the right credentials.

Recent example:

- Pipeline: `galoy-infra`
- Job: `gcp-testflight-postgresql`
- Failed build: `24`
- Stale lock path: `gs://testflight00348-tf-state/testflight00348/postgresql/default.tflock`
- Lock generation / ID: `1779100500648124`

The failure mode was:

```text
Error acquiring the state lock
writing "gs://testflight00348-tf-state/testflight00348/postgresql/default.tflock" failed:
googleapi: Error 412 ... conditionNotMet
```

Clearing it manually required using pipeline credentials from an intercepted Concourse task. That works, but it is awkward and widens the operational access surface.

## Request

Add a Drua operational tool for deleting stale Terraform/OpenTofu lock objects for staging/testflight resources, without requiring direct local `gcloud` access from Codex.

Possible shape:

- Tool name like `gcp_delete_tf_lock`, `staging_delete_tf_lock`, or `terraform_delete_state_lock`.
- Inputs should identify the environment/resource safely, for example:
- project or environment, e.g. `galoy-infra-testflight`
- bucket or testflight UID, e.g. `testflight00348`
- state path/module, e.g. `postgresql`
- optional expected lock generation/ID for compare-and-delete semantics
- The tool should inspect and return lock metadata before deleting:
- lock object path
- generation
- created timestamp
- lock JSON fields: `ID`, `Operation`, `Who`, `Created`, `Path`
- Prefer deleting by exact generation when provided, e.g. `default.tflock#`, so a new legitimate lock cannot be removed accidentally.
- The response should clearly report `deleted`, `not_found`, or `generation_mismatch`.

## Safety requirements

- Restrict this to known staging/testflight state buckets or an allowlist configured in Drua.
- Do not support arbitrary GCS object deletion.
- Require exact object path construction from allowlisted inputs, or require the object path to match a strict `*/default.tflock` pattern under allowed buckets.
- Ideally check recent Concourse job status or lock age before deletion, or at least surface enough metadata for the agent/operator to make the decision.

## Why Drua

Drua already brokers operational tools such as Concourse build inspection behind a controlled interface. This would keep cleanup of stale staging OpenTofu locks inside Drua's controlled tool boundary instead of granting local `gcloud` access to Codex just to unblock CI.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing Drua's existing Concourse build inspection tools and the controlled credential boundary they use. Trace how staging/testflight resources and GCS access are represented, then define the allowlist, exact lock-path construction, metadata inspection, and compare-and-delete behavior. Done means the tool safely reports deleted, not_found, or generation_mismatch without enabling arbitrary GCS deletion.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, rust, terraform
Domain
cloud, devops, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.