dotnet / dotnet/runtime

[API Proposal]: Add `Encoding.TryDetectFromUnicodePreamble` to centralize encoding detection accross .NET ecosystem

Open
#122,236 6 comments 0 reactions 0 assignees View on GitHub
api-suggestion area-System.Text.Encoding
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

### Background and motivation

.NET already exposes functionality of encoding detection based on first few bytes in `StreamReader` class:
https://github.com/dotnet/runtime/blob/2c911651ab05a78ca58f02777b34302ae3a2e07e/src/libraries/System.Private.CoreLib/src/System/IO/StreamReader.cs#L471
This functionality is tightly coupled with `StreamReader`. I think, it can be extracted into its own API, so developers won't need to write their own implementations of various precision/complexity

### API Proposal

```cs
namespace System.Text;

public class Encoding
{
public static bool TryDetectFromUnicodePreamble(ReadOnlySpan input, out Encoding? encoding);
}
```

### API Usage

```csharp
string DecodeUserProvidedBytes(ReadOnlySpan bytes)
{
var encoding = Encoding.TryDetectFromUnicodePreamble(bytes, out var detectedEncoding)
? detectedEncoding
: Encoding.UTF8; // some sane default guess
return encoding.GetString(bytes);
}
```

### Alternative Designs

Make an API, that takes a fallback encoding, e.g. `Encoding Encoding.DetectFromUnicodePreamble(ReadOnlySpan input, Encoding fallbackEncoding)`. The problems I see here are:
- Should the fallback variant be nullable? If yes, what should be the behavior:
- Let's say, throw `ArgumentNullException` if detect didn't succeed. This is bad because now we are throwing argument validation exception based on user input, which is not deterministic
- Ok, maybe always check it for null. Let's say, bytes are not just any user input, but rather come from the source with deterministic rules, so the API user knows that detect based on first few bytes always succeeds and can only return different encodings. Now we are forsing user to pass a non-null dummy unused value
- Don't check fallback case for null at all. But that would mean we have to return a nullable value from the method, which defeats its original purpose (call and always get a result)
- Such shape would limit the fallback behavior user can provide. For instance, let's say user is working on some kind of text editor app and when encoding cannot be determined they want the app user to select one from a combo box. With this API shape such behavior would not be possible without hacks

### Risks

- With the addition of this API .NET runtime now takes responsibility to detect as many common encodings as possble. Given that this functionality already exists in the runtime, I don't consider that risk to be huge

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.