Assert failure in CastHelpers.IsNullableForType for an open Nullable<T>
- Dominant language
- C#
- Stars
- 18.3k
- Forks
- 5.6k
- PR merge metrics
- PR metrics pending
Description
### Description
I'm back with another assert that I've hit while running checked builds. This time it looks like the assert itself might be checking the wrong thing.
`CastHelpers.IsNullableForType` asserts when `typeMT` is an *open* `Nullable` (the generic type definition). This is reachable from ordinary managed code, for example any `IsAssignableFrom` where the destination type is `typeof(Nullable<>)`, so a checked/Debug runtime trips on input that the release runtime handles fine.
The offending line is [`CastHelpers.cs#L598`](https://github.com/dotnet/runtime/blob/main/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/CastHelpers.cs#L598):
```csharp
internal static bool IsNullableForType(MethodTable* typeMT, MethodTable* boxedMT)
{
if (!typeMT->IsNullable)
return false;
Debug.Assert(typeMT->InstantiationArg0() == **typeMT->PerInstInfo); // <-- here
MethodTable* pMTNullableArg = **typeMT->PerInstInfo;
...
}
```
`MethodTable.InstantiationArg0()` is an FCall to `MethodTableNative::InstantiationArg0`, which does:
```cpp
return mt->GetInstantiation()[0].AsMethodTable();
```
and `TypeHandle::AsMethodTable()` asserts `!IsTypeDesc()`. For an open `Nullable`, instantiation argument 0 is a `TypeVarTypeDesc`, not a `MethodTable`, so the assert fires.
The assert is the only thing on this path that calls `InstantiationArg0()`. The line below it reads `**typeMT->PerInstInfo` directly, and that stays valid for an open generic: it gives back a pointer that never matches `boxedMT`, so the method returns `false` like it should. As far as I can tell the problem is only in the diagnostic and release builds are unaffected.
### Reproduction Steps
On a checked/Debug CoreCLR:
```csharp
Console.WriteLine(typeof(int?).IsAssignableFrom(typeof(int))); // control
Console.WriteLine(typeof(Nullable<>).IsAssignableFrom(typeof(int))); // asserts
```
Path in: `RuntimeType.IsAssignableFrom` → `TypeHandle.CanCastToForReflection` → `CanCastToWorker` → `CastHelpers.IsNullableForType`.
### Expected behavior
Prints `True` then `False` without asserting. `typeof(Nullable<>).IsAssignableFrom(x)` is a legal call that works fine in release.
### Actual behavior
The first line prints `True`. The second one asserts:
```
ASSERT FAILED
Expression: !IsTypeDesc()
Location: line 74 in .../src/coreclr/vm/typehandle.inl
Function: AsMethodTable
```
### Regression?
Not sure. Claude thinks it was [#109135](https://github.com/dotnet/runtime/pull/109135) but I've been lied to before.
### Known Workarounds
Use release instead of a checked or debug build.
### Configuration
- .NET 10 (observed on `10.0.10-dev`)
- Reproduced on Windows x64, macOS arm64 and Linux x64, so it doesn't look architecture or OS specific
### Other information
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.