dotnet / dotnet/docs

[Breaking change]: In .NET 8 Stream Reader emits Unicode replacement character, .NET 7 did not

Open
#38,262 4 comments 0 reactions 1 assignee Claimed by @gewarren View on GitHub
:checkered_flag: Release: .NET 8 :watch: Not Triaged breaking-change
Dominant language
No language data
Stars
4.8k
Forks
6.1k
Avg merge
19h 10m
Merged PRs (30d)
268

Description

### Description

When a `StreamReader` with default constructor (UTF-8) encounters a UTF-8 character that is broken in half (one particular kind of invalid UTF-8 byte sequence), the handling changed from .NET 7 to .NET 8. I wasn't able to find docs mentioning this change.

Repro code:
```csharp
using System.Runtime.InteropServices;
using System.Text;
using System.Text.Json;

var str = " \u00B7 ";
var bytes = Encoding.UTF8.GetBytes(str);
Console.WriteLine("Framework: " + RuntimeInformation.FrameworkDescription);
for (var i = 1; i <= bytes.Length; i++)
{
var range = bytes[0..i];
var readByStreamReader = new StreamReader(new MemoryStream(range)).ReadToEnd();
Console.WriteLine(JsonSerializer.Serialize(readByStreamReader));
}
```

Output in .NET 7 (no replacement character emitted):
```
Framework: .NET 7.0.14
" "
" "
" "
" \u00B7"
" \u00B7 "
" \u00B7 "
```

Output in .NET 8 (replacement character emitted)
```
Framework: .NET 8.0.0
" "
" "
" \uFFFD"
" \u00B7"
" \u00B7 "
" \u00B7 "
```

### Version

.NET 8 GA

### Previous behavior

I noticed this on .NET 8 GA. I did not test .NET 8 previews.

### New behavior

A `\uFFFD` character (Unicode replacement character) is emitted by the `StreamReader` now. Previously nothing was emitted.

### Type of breaking change

- [ ] **Binary incompatible**: Existing binaries may encounter a breaking change in behavior, such as failure to load or execute, and if so, require recompilation.
- [ ] **Source incompatible**: When recompiled using the new SDK or component or to target the new runtime, existing source code may require source changes to compile successfully.
- [X] **Behavioral change**: Existing binaries may behave differently at run time.

### Reason for change

Product team can provide details I think.

### Recommended action

Document the change.

### Feature area

Globalization

### Affected APIs

`System.IO.StreamReader`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.