dotnet / dotnet/runtime

[NativeAOT] BestFitMapping tests fail for StringBuilder and layout classes, and crash for string arrays

Open
#133,395 1 comment 0 reactions 0 assignees View on GitHub
area-NativeAOT-coreclr disabled-test untriaged
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

### Description

The five assembly variants under `src\tests\Interop\PInvoke\BestFitMapping` are disabled on NativeAOT using the old dotnet/runtimelab#155 link (now dotnet/runtime#69919).

Temporarily enabling them revealed StringBuilder failures. Temporarily bypassing only the StringBuilder cases then exposed additional layout-class failures and a process crash while testing string arrays. This issue tracks the full test so all of these cases can be investigated before enabling it.

### Reproduction

Observed on Windows x64 with a Checked NativeAOT product build and the `en-US` test path, based on dotnet/runtime commit fb98b574c9c5bcea55f33dbad31c418c8815ce8e, with subsequent local interop-test enablement changes. No regression range has been established.

1. Remove the NativeAOT `ActiveIssue` from `src\tests\Interop\PInvoke\BestFitMapping\Program.cs`.
2. Build and run:

```powershell
src\tests\build.cmd nativeaot checked tree Interop priority 1
src\tests\run.cmd runnativeaottests checked
```

3. To expose the failures after StringBuilder, temporarily bypass `Test.ValidateStringBuilder` in `src\tests\Interop\PInvoke\BestFitMapping\Test.cs` on NativeAOT, and repeat the build and run. This diagnostic bypass should not be required for the eventual fix.

### Observed failures

#### 1. Byref StringBuilder behavior

All five assembly variants fail in `Test.ValidateStringBuilder`. For example:

```text
Assert.NotEqual() Failure: Strings are equal
Expected: Not "\\//::"
Actual: "\\//::"
at Xunit.Assert.NotEqual[T](T, T, IEqualityComparer`1)
at Test.Validate[T,U](Boolean, Boolean, Test.Functions`1, Test.DataContext`2)
at Test.ValidateStringBuilder(Boolean, Boolean, Test.Functions`1)
at Char.PInvoke_Default.RunTest(Boolean, Boolean)
at Program.TestEntryPoint()
```

The assertion compares the original and returned contents after an `[In, Out] ref StringBuilder` call. NativeAOT updates the original builder rather than replacing it, so both references observe the converted contents. The false/false variant similarly reports equal `"?????"` strings.

#### 2. Byref layout-class behavior

With only StringBuilder validation bypassed, `Assembly_Default` and `Assembly_False_False` pass their char/direct-string checks and reach `LPStr.PInvoke_Default.RunTest`. They fail while validating `LPStrTestClass`:

```text
---- Marshal String
---- Marshal String[]
---- Marshal LPStrTestStruct
---- Marshal LPStrTestClass
Xunit.Sdk.NotEqualException: Assert.NotEqual() Failure: Strings are equal
Expected: Not "\\//::"
Actual: "\\//::"
at Xunit.Assert.NotEqual[T](T, T, IEqualityComparer`1)
at Test.Validate[T,U](Boolean, Boolean, Test.Functions`1, Test.DataContext`2)
at LPStr.PInvoke_Default.RunTest(Boolean, Boolean)
at Program.TestEntryPoint()
```

The existing test expects the original reference object to remain unchanged while the byref result contains the marshalled value.

#### 3. String-array crash

In the same run, `Assembly_False_True` gets through the char tests and the direct LPStr string checks, then terminates the merged runner while validating `string[]`:

```text
Validating LPStr marshalling...
-- Validate P/Invokes: BestFitMapping not set, ThrowOnUnmappableChar not set
---- Marshal String
---- Marshal String[]
Expected: 100
Actual: -1073741819
END EXECUTION - FAILED
```

`-1073741819` is `0xC0000005` (access violation). This configuration has assembly-level `BestFitMapping(false, ThrowOnUnmappableChar = true)`. Invalid/unmappable input should result in the expected managed exception, not terminate the process.

The exact crashing instruction has not been identified. One source-level investigation lead is array exception cleanup: NativeAOT's `ArrayMarshaller` leaves input buffers uninitialized and emits cleanup over the full array length, including when an element conversion throws before initialization completes.

### Related issues / duplicate check

- dotnet/runtimelab#172 records the earlier StringBuilder `AreNotEqual` failure in this test family.
- dotnet/runtime#123529 tracks StringBuilder marshalling more generally.
- dotnet/runtime#81674 is closed and concerns other NativeAOT array cases, including empty arrays and out-only buffers; the crash above occurs in the best-fit/unmappable-character test path.
- dotnet/runtime#33950 is a closed CoreCLR GC-stress/JIT struct-marshaler issue, rather than this NativeAOT run.

Searches for BestFitMapping, ThrowOnUnmappableChar, LPStrTestClass, NativeAOT array marshalling, and layout-class marshalling did not find an existing report matching the newly observed layout-class failure and string-array crash. The StringBuilder overlap is noted above.

> [!NOTE]
> This issue was generated by GitHub Copilot from a local investigation.

Contributor guide

Open the contributing guide

Research direction

Start with src\tests\Interop\PInvoke\BestFitMapping\Program.cs and Test.cs, then reproduce with the listed NativeAOT build and run commands. Compare the StringBuilder, layout-class, and string-array paths, including ArrayMarshaller cleanup, while preserving the diagnostic bypass only for investigation. Done means all five assembly variants run without bypasses, assertion failures, or process crashes.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
operating-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.