dotnet / dotnet/runtime

HttpWebResponse: CharacterSet on "Content-Type: text/event-stream" is mapped to "ISO-8859-1" but should be "UTF-8" by specification

Open
#131,218 3 comments 0 reactions 0 assignees View on GitHub
area-System.Net.Http
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

### Description

The SSE Specification (https://html.spec.whatwg.org/multipage/server-sent-events.html) states that the Content-Type is "text/event-stream" and the encoding is always UTF-8.

The CharacterSet property of the HttpWebResponse falls back to ISO-8859-1 in this case.

See:
https://github.com/dotnet/runtime/blob/57f3e7eaec55cf263674f65eabb8965b612dc997/src/libraries/System.Net.Requests/src/System/Net/HttpWebResponse.cs#L278

### Reproduction Steps
``` csharp

// Repro.cs
//
// Bug: HttpWebResponse.CharacterSet returns "ISO-8859-1" as a silent default for ANY
// text/* Content-Type that does not declare an explicit charset parameter, instead of
// returning null/empty. This causes actually-UTF-8-encoded response bodies (e.g. from
// text/event-stream SSE endpoints, which commonly omit charset) to be mis-decoded if a
// caller trusts CharacterSet directly.
//
// Run: dotnet run
using System;
using System.IO;
using System.Net;
using System.Text;
using System.Threading.Tasks;

class Repro
{
static async Task Main()
{
// --- minimal local server: no charset declared, body actually UTF-8 ---
string prefix = "http://localhost:8085/";
using var listener = new HttpListener();
listener.Prefixes.Add(prefix);
listener.Start();

var serverTask = Task.Run(async () =>
{
HttpListenerContext ctx = await listener.GetContextAsync();
byte[] body = Encoding.UTF8.GetBytes("caf\u00e9 \u00fc\u00f6\u00e4 \u2192"); // "café üöä →"
ctx.Response.ContentType = "text/event-stream"; // NOTE: no "; charset=utf-8"
ctx.Response.ContentLength64 = body.Length;
await ctx.Response.OutputStream.WriteAsync(body, 0, body.Length);
ctx.Response.OutputStream.Close();
});

// --- client: exactly what APIClientRequest.cs does ---
var request = (HttpWebRequest)WebRequest.Create(prefix);
using var response = (HttpWebResponse)await request.GetResponseAsync();

Console.WriteLine($"Content-Type header sent by server : {response.ContentType}");
Console.WriteLine($"HttpWebResponse.CharacterSet : \"{response.CharacterSet}\"");
// BUG: .NET returns "ISO-8859-1" here even though NO charset was ever declared,
// per an old RFC 2616 default for text/* - the server actually sent UTF-8 bytes.

Encoding wrongEncoding = Encoding.GetEncoding(response.CharacterSet);
using var stream = response.GetResponseStream();
using var reader = new StreamReader(stream, wrongEncoding);
string mangled = await reader.ReadToEndAsync();

Console.WriteLine($"Decoded using CharacterSet ({wrongEncoding.EncodingName}): {mangled}");
Console.WriteLine($"Expected (decoded as UTF-8) : café üöä →");

listener.Stop();
await serverTask;
}
}

```

### Expected behavior

CharacterSet on Requests with "Content-Type: text/event-stream" should stay with the default UTF-8 encoding.

### Actual behavior

CharacterSet on Requests with "Content-Type: text/event-stream" is reported as "ISO-8859-1".

### Regression?

_No response_

### Known Workarounds

- Do not use CharacterSet at all and try to find the text encoding yourself.
- If you could change the Serverside: append "; charset=UTF-8" to the "Content-Type:" Header

that would to the trick:

``` csharp
if (srchString.AsSpan().Trim().StartsWith("text/", StringComparison.Ordinal)
&& !srchString.AsSpan().Trim().Equals("text/event-stream", StringComparison.Ordinal))
{
_characterSet = "ISO-8859-1";
}
```
### Configuration

dotnet 10.0.300, Win11, x64

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in src/libraries/System.Net.Requests/src/System/Net/HttpWebResponse.cs at the CharacterSet logic referenced by the issue. Run the provided Repro.cs with the local text/event-stream response and inspect the existing System.Net.Requests tests for related CharacterSet behavior. Done means the response reports the specification-compatible encoding behavior and the UTF-8 sample is not mis-decoded.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.