mudler / mudler/vllm.cpp

test_opt_paged_engine: removing greedy_ids.npy deletes the whole SACRED gate into a green at 0 assertions

Open
#2,988 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: MODEL-TEXT-opt-optfor-causal-lm

#2805 / #2981 closed one mute switch in
tests/vllm/models/test_opt_paged_engine.cpp. Two of the same class survive in
the same file, and one of them is worse than the one that was fixed.

1. Removing greedy_ids.npy deletes the whole SACRED gate into a green

Measured by a fresh reviewer of #2981, on both the pre-fix and post-fix
binaries:

assertions: 0 | 0 passed | 0 failed | Status: SUCCESS!    rc 0

The bar itself is behind an existence guard, so removing one committed file
makes the entire token-exact gate assert nothing and report success. The ctest
entry at tests/CMakeLists.txt:3663-3667 carries no
FAIL_REGULAR_EXPRESSION, and the case never returns 77, so nothing outside
the binary notices either.

.agents/specs/sweep-opt-125m.md currently calls this sibling guard "already
correct". That is too strong: it asserts nothing rather than asserting
something unlicensed, which is less bad, not correct. assertions: 0 is a
skip wearing a pass.

2. p3_prompt.i32 absence silently drops a cross-check

Removing it takes the post-fix binary from 43 to 42 assertions, still
SUCCESS, rc 0 -- if (!want_prompt.empty()) CHECK(out.prompt_token_ids == want_prompt) at :304. Same absence-means-skip shape. Pre-existing, inside
the region #2981 left byte-unchanged, so neither introduced nor claimed fixed
there.

What closes this

The golden set is a precondition of the gate, not an option: a missing
greedy_ids.npy or prompt file reds rather than reducing the assertion count.
Red-before for each: remove the file, confirm the gate reds where it currently
passes. A FAIL_REGULAR_EXPRESSION or an assertion-count floor on the ctest
entry would also catch the assertions: 0 shape from outside.

Related: #2987 (the same licence defect in four sibling gates).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in tests/vllm/models/test_opt_paged_engine.cpp, especially the greedy_ids.npy guard and the p3_prompt.i32 check around line 304, then inspect the ctest entry in tests/CMakeLists.txt:3663-3667. Remove each golden file to confirm the current green behavior, then make the gate fail when a required file is absent or assertions drop to zero; verify both missing-file cases turn red.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.