dotnet / dotnet/runtime

[API Proposal]: New StreamReader Property CurrentBOM

Open
#128,267 4 comments 0 reactions 0 assignees View on GitHub
api-suggestion area-System.IO
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

### Background and motivation

The new API purpose is creation of a new property of [StreamReader](https://learn.microsoft.com/en-us/dotnet/api/system.io.streamreader?view=netframework-4.8.1#properties) called CurrentBOM, which allows one to determine the presence of a [byte order mark (BOM)](https://learn.microsoft.com/en-us/dotnet/api/system.text.encoding.getpreamble?view=netframework-4.8.1#remarks) via boolean result. This is due to how the present behavior of [CurrentEncoding](https://learn.microsoft.com/en-us/dotnet/api/system.io.streamreader.currentencoding?view=netframework-4.8.1#system-io-streamreader-currentencoding) is [not designed](https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/IO/StreamReader.cs) to properly detect existence of a BOM in UTF encoded files.

In addition, approving this proposal avoids the .NET+PWSH community headaches when interoperating with UTF files at scale.

Specifically:
1) With C# [reading files line by line](https://stackoverflow.com/questions/5012167/how-can-i-detect-if-a-net-streamreader-found-a-utf8-bom-on-the-underlying-strea) in an attempt to detect existence of a BOM. This approach is wasteful in comparison to the proposed, and less efficient speedwise then [ReadToEnd()](https://learn.microsoft.com/en-us/dotnet/api/system.io.streamreader.readtoend?view=netframework-4.8.1&viewFallbackFrom=net-10.0).
2) With .NETized PWSH, it's being forced to do concessionary workarounds (e.g. reading twice, reading once and doing byte conversions in-memory to do a manual custom one-off check, etc.).

Example PWSH code that demonstrates the problem below:
```
$bompath="$env:TEMP\bom.txt"
$utf8Bom=[System.Text.UTF8Encoding]::new($true)
[System.IO.File]::WriteAllText($bompath,'',$utf8Bom)
$bytes=[System.IO.File]::ReadAllBytes($bompath)
if($bytes.Length -eq 3 -and $bytes[0] -eq 0xEF -and $bytes[1] -eq 0xBB -and $bytes[2] -eq 0xBF){
write-host 'BOM DETECTED'
}else{
write-host 'NO BOM DETECTED'
}
$sr=New-Object System.IO.StreamReader($bompath,$false)
$encoding=$sr.CurrentEncoding
$encoding.GetPreamble().length

$nobompath="$env:TEMP\bom-no.txt"
$utf8NoBom=[System.Text.UTF8Encoding]::new($false)
[System.IO.File]::WriteAllText($nobompath,'',$utf8NoBom)
$bytes=[System.IO.File]::ReadAllBytes($nobompath)
if($bytes.Length -eq 3 -and $bytes[0] -eq 0xEF -and $bytes[1] -eq 0xBB -and $bytes[2] -eq 0xBF){
write-host 'BOM DETECTED'
}else{
write-host 'NO BOM DETECTED'
}
$sr=New-Object System.IO.StreamReader($nobompath,$false)
$encoding=$sr.CurrentEncoding
$encoding.GetPreamble().length
```

What is expected:
```
BOM DETECTED
3
NO BOM DETECTED
0
```

What you get:
```
BOM DETECTED
3
NO BOM DETECTED
3
```

### API Proposal

```csharp
namespace System.IO;

public partial class StreamReader
{
public bool CurrentBOM { get; }
}
```

### API Usage

```csharp
using System;
using System.IO;
using System.Text;

class Test
{

public static void Main()
{
string path = @"c:\temp\MyTest.txt";

try
{
if (File.Exists(path))
{
File.Delete(path);
}

//Use UTF-16 encoding
using (StreamWriter sw = new StreamWriter(path, false, new UnicodeEncoding()))
{
sw.WriteLine("My test");
sw.WriteLine("text.");
}

using (StreamReader sr = new StreamReader(path, true))
{
while (sr.Peek() >= 0)
{
Console.Write((char)sr.Read());
}

//Test for BOM after reading, or at least after the first read.
Console.WriteLine("BOM present: {0}.", sr.CurrentBOM);
}
}
catch (Exception e)
{
Console.WriteLine("The process failed: {0}", e.ToString());
}
}
}
```

### Alternative Designs

One could amend the [CurrentEncoding Property](https://learn.microsoft.com/en-us/dotnet/api/system.io.streamreader.currentencoding?view=netframework-4.8.1#system-io-streamreader-currentencoding) such that when performing a GetPreamble() the length is properly detected (i.e. proper endian, number of bytes, and type of bytes in precise order). While doing so would introduce a breaking change (i.e. a feature not implemented accurately is now accurate and precise) having accuracy and precision on the logic is reasonable.

### Risks

None identified.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.