stacklok / stacklok/mecatl

test: diagnose intermittent JSONL inventory catalog preparation failure

Open
#1,223 0 comments 0 reactions 1 assignee View on GitHub

@JAORMX is already working on this.

Since Sep 8, 2026.

Dominant language
Go
Stars
152
Forks
16
Avg merge
14h 48m
Merged PRs (30d)
536

Description

Problem

TestUsableAutoSkillsStockBuildPolicyMatrix/explicit_evaluated failed on an otherwise unrelated CI-only perf-workflow PR with:

jsonlstore: inventory changed repeatedly while preparing catalog

Observed in PR #1222, CI run 34153750204. The PR changes only .github/workflows/perf.yml, so it cannot directly change the learning or JSONL-store execution path.

Relevant path

Store.readyInventoryCatalog makes up to three attempts to observe a stable inventory fingerprint around discoveryMetaListLocked, then returns the error above if the inventory keeps changing:

  • internal/adapter/store/jsonlstore/metalist.go:281-310
  • internal/app/usable_auto_skills_matrix_e2e_test.go (TestUsableAutoSkillsStockBuildPolicyMatrix)

The test's diagnostics show the stale-session sweeper's list operation failed with this error, after which the expected learned skill was absent.

Related work

#1088 tracks a different intermittent failure in the same matrix test (learned skill operation failed) and suspected skill-store publication contention. This report records the distinct, actionable JSONL inventory-catalog failure path.

Acceptance criteria

  • Reproduce or otherwise identify the concurrent inventory mutation that exhausts the three-attempt catalog preparation loop during this test.
  • Determine whether the retry policy is insufficient or a test/production synchronization defect exists.
  • Preserve fail-closed cursor and inventory consistency semantics.
  • Add a deterministic regression test once the source is identified.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.