Runtime JIT compilation contaminates published TTFT and TPOT measurements

Start with workflow run 30933463657, attempt 2, and its worker logs; search for “JIT compilation during inference” and compare timestamps with the C12 measurement boundary. Trace benchmark config 2202 and the affected benchmark IDs through measurement, CI validation, DB export, and dashboard aggregation. Done means runtime compiler events are detected, affected rows are invalidated or flagged, and exported records retain the event details.

Describe the bug

InferenceX can publish TTFT and TPOT measurements while workers are still compiling kernels. These rows mix cold-start compilation with steady-state inference, but the DB dump does not mark them as contaminated.

The latest stable dump still shows the problem:

• Release: db-dump/2026-08-31
• Published: 2026-08-31
• 83,561 benchmark rows; 8,342 latest single-turn operating points
• 357 strong-signal TTFT-tail candidates and 17 strong-signal TPOT-tail candidates
• 193 TTFT and 1 TPOT candidates recur enough to be high-priority checks
• The candidate set is unchanged from db-dump/2026-08-24

Tail statistics alone do not prove JIT. The MiniMax M3 example below is
log-confirmed; the DeepSeek-R1 example is a persistent regression correlated
with warmup/cache changes.

Reproduction: log-confirmed MiniMax M3

Workflow: run 30933463657, attempt 2

Artifact: multinode_server_logs_minimaxm3_8k1k_fp4_dynamo-vllm_prefill-tp4-pp1-dcp1-pcp1-ep1-dpfalse-nw1_decode-tp4-pp1-dcp1-pcp1-ep1-dpfalse-nw6_disagg-true_spec-mtp_conc12x24x48_b200-dgxc_07

Configuration:

• DB config ID 2202; benchmark IDs 438518/438520/438519
• MiniMax M3 FP4, B200, Dynamo-vLLM, MTP/EAGLE3
• ISL 8192, OSL 1024
• One TP4 prefill worker and six TP4 decode workers
• C12 process: 18:38:17–18:40:37 UTC
• Unmeasured probe: 18:38:32.17–18:38:42.79 UTC
• Main C12 request burst begins at 18:38:42.81 UTC
• The server reports enable_jit_warmup=True

Search the worker logs for JIT compilation during inference. There are 50
warnings: 37 Triton and 13 CuTeDSL. Thirty-four occur after the main C12
request burst begins.

All log-confirmed JIT sites

Correlating frontend routing/completion logs with each worker's JIT timestamps
identifies all ten first-batch requests whose assigned decode worker compiled
before their first token. The two requests assigned to already-warmed
decode_w1 are shown as controls.

The confirmed victim is the C12 benchmark record and its TTFT, TPOT, ITL, and
E2EL distributions.

C24 and C48 also have TPOT tails, but no later JIT warning appears. They should
not be attributed to JIT without trace evidence.

Reproduction: persistent DeepSeek-R1 TTFT victim

Configuration: DSR1 FP4, B200, SGLang, ISL 1024, OSL 8192, TP4, PP1,
attention DP1, MoE EP1, MoE TP4.

The older server artifacts have expired, so this case is probable rather than
log-confirmed. The regression starts immediately after:

• dd7e3850 removed explicit DSR1 JIT warmup.
• 15b7c00b removed the persistent FlashInfer cache.
• 17d5e204 changed the model to the V2 name; the launch script still matched only the old name, reducing 1k/8k measured requests from CONC * 20 to CONC * 10.

Before the changes, C32 TTFT was 464 ms mean / 1.11 s P99. It became 3.77 s /
68.92 s on 2025-10-07 and remains 7.88 s / 75.26 s in the latest selected row.

Expected behavior

• Warmup covers every prefill/decode worker and every measured shape/config.
• Measurement starts only after all workers complete warmup.
• A runtime-JIT warning after the measurement boundary invalidates or clearly
  flags the affected benchmark row.
• The dump exports JIT events or a contamination flag before Actions artifacts
  expire.
• Dashboard means exclude or visibly mark contaminated samples.

Acceptance criteria

• Rerunning MiniMax M3 config 2202 produces no runtime-JIT warning after the measurement boundary.
• Rerunning the DSR1 configuration removes the recurring C32/C64/C128 TTFT cold tail.
• CI fails or marks a benchmark invalid when Triton, CuTeDSL, FlashInfer, or another monitored compiler runs during measurement.
• DB exports retain the event type, target, worker, timestamp, and affected benchmark ID.
• C24/C48 MiniMax M3 TPOT tails are traced separately instead of being assumed to be JIT.