ci_settings k8s_grace_period should not apply when jobs have reached their backoff limit

Start with the provided Tiltfile and job.yaml, then run the tilt ci reproduction with ci_settings(k8s_grace_period='30s'). Trace the Kubernetes job and pod grace-period handling, including why the observed wait is five minutes. Done means a job that reaches its backoff limit fails without waiting for the grace period.

Expected Behavior

If a job has reached its backoff limit, Tilt needn't wait around for the k8s_grace_period to expire.

Current Behavior

The job fails and tilt ci hangs until the grace period expires.

Steps to Reproduce

Tiltfile:

ci_settings(k8s_grace_period='30s')
k8s_yaml('job.yaml')

job.yaml:

apiVersion: batch/v1
kind: Job
metadata:
  name: always-fails
spec:
  template:
    spec:
      containers:
      - name: always-fails
        image: busybox
        command:
          - sh
          - -c
          - echo "hardcoded failure" && exit 1
      restartPolicy: Never
  backoffLimit: 0

Output:

[matt.terwilliger@D94GKM7XNP tilt]$ time tilt ci
Tilt started on http://localhost:10350/
v0.31.1, built 2023-01-09

Initial Build
Loading Tiltfile at: /private/tmp/tilt/Tiltfile
Successfully loaded Tiltfile (4.6295ms)
 always-fails │
 always-fails │ Initial Build
 always-fails │ STEP 1/1 — Deploying
 always-fails │      Applying YAML to cluster
 always-fails │      Objects applied to cluster:
 always-fails │        → always-fails:job
 always-fails │
 always-fails │      Step 1 - 0.46s (Deploying)
 always-fails │      DONE IN: 0.46s
 always-fails │
 always-fails │
 always-fails │ Tracking new pod rollout (always-fails-924fb):
 always-fails │      ┊ Scheduled       - <1s
 always-fails │      ┊ Initialized     - <1s
 always-fails │      ┃ Not Ready       - (ContainersNotReady): containers with unready status: [always-fails]
 always-fails │ [event: pod ws-matt-terwilliger/always-fails-924fb] Pulling image "busybox"
 always-fails │ [event: pod ws-matt-terwilliger/always-fails-924fb] Successfully pulled image "busybox" in 1.712989824s
 always-fails │ [event: job ws-matt-terwilliger/always-fails] Job has reached the specified backoff limit
 always-fails │ hardcoded failure
Error: exceeded grace period: Pod "always-fails-924fb" failed
tilt ci  0.99s user 0.49s system 0% cpu 5:20.22 total

There may be something else going on as well? I'm not sure why this took 5 minutes and not my 30s specified grace period.

About Your Use Case

Using tilt ci to run a series of smoke tests in CI.