microsoft / microsoft/DirectXShaderCompiler

Validation: local noinline functions in library treated as export functions

Open
#7,152 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug validation
Dominant language
C++
Stars
3.7k
Forks
900
Avg merge
2d 11h
Merged PRs (30d)
44

Description

DxilValidation will treat [noinline] functions as if they are exported functions, even though they are local, and rules should be different (depending on what calls them).

Currently, operations that are legal in specific shader stages, but not export functions, are allowed in [noinline] local functions called by the main entry function when compiling to a non-library target. However, these same operations will fail validation when compiling the same code to a library, due to the treatment of [noinline] functions as if they are export functions.

One way to resolve this discrepancy is to tighten the validation for [noinline] functions for non-library shader targets, by considering them to be like export functions in this context as well. This will start failing code that is currently considered legal. However, [noinline] usage should be rare, and is already considered somewhat experimental and use at your own risk due to known bugs here.

If we wanted to fix this for libraries, we could do so by constructing a mask of the types of entry (or export) functions that call each internal function, then use that mask to check legal intrinsic usage. This same mask approach is already used to construct export function compatibility masks in the RDAT part.

This shader succeeds if compiled to ps_6_8, but fails if you compile to lib_6_8 due to this issue:
https://godbolt.org/z/eWYcrn48W

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the linked Godbolt example for ps_6_8 and lib_6_8, then start at DxilValidation and compare its handling of local [noinline] functions with export functions. Review the RDAT export-function compatibility-mask approach as a possible guide. Done means validation behavior is consistent for the two target types, with the intended rules for functions called by entry or export functions.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers, computer-graphics
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.