intel / intel/torch-xpu-ops

[Bug] TestDropoutAlignRandomEager.test_dropout_mask_parity_and_rng_offset_cuda: eager-vs-compiled dropout mask mismatch ~50% on XPU (recurrence of #4851)

Open
#4,882 1 comment 0 reactions 1 assignee View on GitHub

@jkosnox is already working on this.

Since Aug 10, 2026.

bug module: inductor test: ut
Dominant language
Python
Stars
113
Forks
129
Avg merge
5d 9h
Merged PRs (30d)
112

Description

Bug Description

TestDropoutAlignRandomEager.test_dropout_mask_parity_and_rng_offset_cuda
fails on XPU. The test asserts that torch.nn.Dropout run in eager mode and
the same module wrapped with torch.compile produce matching dropout masks
(and matching RNG-offset deltas) when seeded identically. On XPU, only
~49.95% of mask elements agree between eager and compiled output, i.e. the
masks are effectively uncorrelated (random ~50% agreement) rather than
identical, for all three tested dtypes (float32, float16, bfloat16).

Affected Test

op_ut,test/inductor/test_dropout_align_random_eager.py,TestDropoutAlignRandomEager.test_dropout_mask_parity_and_rng_offset_cuda

Note: the test method is only defined with a _cuda suffix in current
upstream pytorch/pytorch source (no _xpu-suffixed or generic variant
exists); it runs on XPU because GPU_TYPE/HAS_GPU_AND_TRITON resolve to
XPU in this environment and the class is gated only by
requires_gpu()/IS_LINUX and HAS_GPU_AND_TRITON, not by an explicit
CUDA-only check.

Reproduce

conda activate pytorch_xpu
cd /tmp   # avoid running from inside the pytorch source checkout (conftest.py double-load issue)
python -m pytest -sv "/home/daisyden/upstream/inductor_agnostic/test/inductor/test_dropout_align_random_eager.py::TestDropoutAlignRandomEager::test_dropout_mask_parity_and_rng_offset_cuda"

Observed Failure (this run)

FAILED test/inductor/test_dropout_align_random_eager.py::TestDropoutAlignRandomEager::test_dropout_mask_parity_and_rng_offset_cuda

Traceback (most recent call last):
  File "test/inductor/test_dropout_align_random_eager.py", line 257, in test_dropout_mask_parity_and_rng_offset_cuda
    self.assertLessEqual(
  File ".../unittest/case.py", line 1238, in assertLessEqual
    self.fail(self._formatMessage(msg, standardMsg))
  File ".../unittest/case.py", line 675, in fail
    raise self.failureException(msg)
AssertionError: 0.5005362033843994 not less than or equal to 0.0001
Dropout mask mismatch ratio too high: 0.50053620

1 failed in 15.03s

Measured mismatch ratio: 0.5005362 (threshold is 1e-4), i.e. eager and
compiled masks agree only at chance level (~50%), for the first dtype
(float32) tested in the loop; the assertion fails before later dtypes in
the loop are reached.

Root Cause Analysis

The test seeds eager and compiled runs identically via _set_seed(BASE_SEED)
and expects Inductor's Philox-based RNG to reproduce the same
stream/offset as the eager XPU RNG so that dropout masks match bit-for-bit.
Instead, the compiled-vs-eager mask agreement (~50%) indicates the two paths
are drawing effectively independent/uncorrelated random streams rather than
sharing the same seed/offset contract on XPU. This points to a divergence
between the XPU eager RNG stream and the Inductor-generated (Philox) RNG
offset/seed handling used for compiled dropout on XPU — the same failure
mode already described in #4851 for a set of related tests in this same
file (including this exact test method and error message).

Context

  • intel/torch-xpu-ops#4851 — "[Bug] TestDropoutAlignRandomEager: compiled
    dropout mask ~50% mismatch vs eager on XPU" was filed for the same failure
    signature (same test method, same AssertionError: ... not less than or equal to 0.0001 / Dropout mask mismatch ratio too high: 0.5005...) across
    4 test methods in this class. That issue was closed on 2026-08-09 without
    a linked commit or merged PR
    (gh api .../issues/4851/timeline shows the
    closed event has commit_id: null), so the closure does not represent a
    verified fix. This new issue is filed because the bug still reproduces
    locally against the current checkout, per the reproduce steps above.
  • pytorch/pytorch#190237 — "TestDropoutAlignRandomEager: 4 tests fail on
    main" (OPEN). Related context on this test class's general flakiness/known
    issues, though it covers different test methods
    (test_linear_block_compile_parity_forward/backward,
    test_dropout_parity_dynamic_shapes) which are already @unittest.skip'd
    in-file citing that issue. Low relevance to the specific RNG-offset root
    cause here but included for completeness since it is the same test class.

Environment

torch: 2.14.0.dev20260726+xpu
Platform: Linux
Python: 3.10.20
GPU: Intel(R) Data Center GPU Max 1550 (device_id=0xBD5)
Driver: 1.6.33578+57 (Level-Zero, oneAPI Unified Runtime)
Checkout: pytorch/pytorch @ deefeb3bd593d7a810baad416abbd9ce180dd98a (branch daisyden/inductor)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.