Shopify / Shopify/ci-queue

UnresolvedEntry leaks to test runner in lazy-load retry runs

Open
#396 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Ruby
Stars
239
Forks
36
Avg merge
28m
Merged PRs (30d)
3

Description

In lazy-load mode, workers on a retry run can end up with resolve_entry returning an UnresolvedEntry instead of a live test object, crashing the test runner:

NoMethodError: undefined method `run' for an instance of CI::Queue::Redis::UnresolvedEntry
    minitest/queue.rb:~167
Root cause

PR #380 introduced UnresolvedEntry as a defense-in-depth fallback for when resolve_entry finds neither @index nor entry_resolver set. The same PR added configure_lazy_queue to the eager-mode branch of populate_queue so non-leader workers get an entry_resolver.

The gap is in the retry join path. When a worker enters a retry run, it can skip populate_queue entirely -- going straight to the retry queue pop loop. configure_lazy_queue is never called in that path, so entry_resolver stays nil. The next test popped off the queue falls through to the UnresolvedEntry fallback. The runner then calls .run on it and crashes.

With many parallel workers this cascades: the unresolved entry is never acknowledged, gets requeued, and each worker that picks it up crashes the same way.

Conditions
  • Lazy-load mode enabled
  • Retry run (automatic retry is the most common trigger in CI environments)
  • Non-leader worker (the leader goes through populate_queue and gets entry_resolver set)
Fix direction

Call configure_lazy_queue in the retry join path, not just inside populate_queue. Mirrors the eager-mode fix from #380 for the initial run.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read minitest/queue.rb around line 167, then trace the retry join path and the existing configure_lazy_queue call in populate_queue. Confirm that non-leader workers in lazy-load retry runs configure an entry_resolver before popping tests, and verify that the runner receives a live test object rather than an UnresolvedEntry.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.