utopia-php / utopia-php/monorepo

KubernetesJob does not extend the ack deadline, so a long job is redelivered while it is still running

Open
#297 0 comments 0 reactions 1 assignee View on GitHub

@levivannoort is already working on this.

Since Sep 18, 2026.

Dominant language
PHP
Stars
3
Forks
4
Avg merge
12h 25m
Merged PRs (30d)
103

Description

Adapter\KubernetesJob does not override withAckExtension(), so a run-to-completion worker gets the
base no-op and its message is never held open. Any handler that outlives ackWait is redelivered while
it is still running
.

// packages/queue/src/Queue/Adapter.php
protected function withAckExtension(Consumer $consumer, Queue $queue, Message $message, \Closure $work): void
{
    $work();
}

Only Adapter\Swoole overrides it, with a heartbeat every extendInterval() (ackWait / 3) for as long
as the handler runs.

Who this affects

worker-migrations is the only worker on this adapter today
(_APP_QUEUE_ADAPTER_MIGRATIONS=kubernetes), and it is the worst possible one to have it: the
application-configuration comment records that the longest observed migration ran three hours, and
the Job carries activeDeadlineSeconds: 28800.

It is not a problem on Redis — the Redis broker has no ack deadline, and a stranded claim is recovered by
the reap sweep instead. It becomes one the moment the queue moves to JetStream, where an unacked message
comes back on ackWait regardless of whether the handler is still working. A redelivered migration is a
second Job running the same migration concurrently.

That is why v1-migrations is held back in
application-configuration#322
while v1-jobs moves: worker-jobs is on the Swoole adapter and does get the heartbeat.

Verified

The heartbeat's effect was measured directly, on a Swoole consumer against fra1 staging — 40s of handler
work against a 30s ackWait:

redelivered
without the heartbeat 2 of 8
with it 0

So the mechanism works where it is wired; it simply is not wired on this adapter.

Either fix works
  • Teach KubernetesJob to extend. It already drives receive/commit/reject through the same
    Consumer interface and is broker-agnostic, so the heartbeat has everything it needs. The adapter is
    sequential rather than coroutine-based, which is the only reason Swoole's implementation does not
    transfer unchanged.
  • Or require ackWait >= activeDeadlineSeconds on any queue using this adapter, and refuse the
    combination at construction the way the broker already refuses a backoff schedule that contradicts
    its ackWait. That turns a silent duplicate into a boot-time refusal.

The second is cheaper and would have caught this; the first is what makes long run-to-completion jobs
actually safe on JetStream.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.