dotnet / dotnet/runtime

Assert failure in CastHelpers.IsNullableForType for an open Nullable<T>

Open
#133,448 1 comment 1 reaction 2 assignees Claimed by @jkotas View on GitHub
area-System.Reflection
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

### Description

I'm back with another assert that I've hit while running checked builds. This time it looks like the assert itself might be checking the wrong thing.

`CastHelpers.IsNullableForType` asserts when `typeMT` is an *open* `Nullable` (the generic type definition). This is reachable from ordinary managed code, for example any `IsAssignableFrom` where the destination type is `typeof(Nullable<>)`, so a checked/Debug runtime trips on input that the release runtime handles fine.

The offending line is [`CastHelpers.cs#L598`](https://github.com/dotnet/runtime/blob/main/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/CastHelpers.cs#L598):

```csharp
internal static bool IsNullableForType(MethodTable* typeMT, MethodTable* boxedMT)
{
if (!typeMT->IsNullable)
return false;

Debug.Assert(typeMT->InstantiationArg0() == **typeMT->PerInstInfo); // <-- here
MethodTable* pMTNullableArg = **typeMT->PerInstInfo;
...
}
```

`MethodTable.InstantiationArg0()` is an FCall to `MethodTableNative::InstantiationArg0`, which does:

```cpp
return mt->GetInstantiation()[0].AsMethodTable();
```

and `TypeHandle::AsMethodTable()` asserts `!IsTypeDesc()`. For an open `Nullable`, instantiation argument 0 is a `TypeVarTypeDesc`, not a `MethodTable`, so the assert fires.

The assert is the only thing on this path that calls `InstantiationArg0()`. The line below it reads `**typeMT->PerInstInfo` directly, and that stays valid for an open generic: it gives back a pointer that never matches `boxedMT`, so the method returns `false` like it should. As far as I can tell the problem is only in the diagnostic and release builds are unaffected.

### Reproduction Steps

On a checked/Debug CoreCLR:

```csharp
Console.WriteLine(typeof(int?).IsAssignableFrom(typeof(int))); // control
Console.WriteLine(typeof(Nullable<>).IsAssignableFrom(typeof(int))); // asserts
```

Path in: `RuntimeType.IsAssignableFrom` → `TypeHandle.CanCastToForReflection` → `CanCastToWorker` → `CastHelpers.IsNullableForType`.

### Expected behavior

Prints `True` then `False` without asserting. `typeof(Nullable<>).IsAssignableFrom(x)` is a legal call that works fine in release.

### Actual behavior

The first line prints `True`. The second one asserts:

```
ASSERT FAILED
Expression: !IsTypeDesc()
Location: line 74 in .../src/coreclr/vm/typehandle.inl
Function: AsMethodTable
```

### Regression?

Not sure. Claude thinks it was [#109135](https://github.com/dotnet/runtime/pull/109135) but I've been lied to before.

### Known Workarounds

Use release instead of a checked or debug build.

### Configuration

- .NET 10 (observed on `10.0.10-dev`)
- Reproduced on Windows x64, macOS arm64 and Linux x64, so it doesn't look architecture or OS specific

### Other information

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.