dotnet / dotnet/aspnetcore

JSON lines support

Open
#67,203 1 comment 0 reactions 0 assignees View on GitHub
api-proposal api-suggestion area-minimal
Dominant language
C#
Stars
38.4k
Forks
10.9k
Avg merge
2d 9h
Merged PRs (30d)
276

Description

## Background and Motivation

[JSON Lines (JSONL)](https://jsonlines.org/) is a text-based format where each line is an independent JSON value, most commonly a JSON object. It is well-suited for incremental processing, large payloads, and streaming scenarios where emitting or consuming a single JSON array would require buffering or delayed completion.

ASP.NET Core already has strong support for JSON request and response handling through:
- MVC input/output formatters
- `HttpRequestJsonExtensions`
- `HttpResponseJsonExtensions`
- minimal API result helpers

However, the existing APIs assume a single JSON document per request or response body. JSON Lines is different in a way that is observable to callers: instead of a single top-level JSON value such as an object or array, the payload is a sequence of top-level JSON values delimited by newlines.

This proposal adds first-class ASP.NET Core HTTP support for JSON Lines, building on the base serialization work tracked by dotnet/runtime#126395.

The primary goals are:

- Enable efficient streaming responses for `IAsyncEnumerable` with a standard media type.
- Enable reading request bodies that contain newline-delimited JSON values.
- Make JSON Lines support explicit and discoverable in minimal APIs.
- Allow MVC and minimal APIs to participate in content negotiation and endpoint metadata using `application/jsonl`.

The proposed shape intentionally keeps JSON Lines separate from existing `ReadFromJsonAsync` and `WriteAsJsonAsync` APIs because JSONL is not just another JSON subtype; it changes framing and payload semantics.

## Proposed API

```diff
namespace Microsoft.AspNetCore.Http;

public static class HttpRequestJsonExtensions
{
+ public static IAsyncEnumerable ReadFromJsonLinesAsAsyncEnumerable(
+ this HttpRequest request,
+ JsonSerializerOptions? options = null,
+ CancellationToken cancellationToken = default);

+ public static IAsyncEnumerable ReadFromJsonLinesAsAsyncEnumerable(
+ this HttpRequest request,
+ Type type,
+ JsonSerializerOptions? options = null,
+ CancellationToken cancellationToken = default);

+ public static IAsyncEnumerable ReadFromJsonLinesAsAsyncEnumerable(
+ this HttpRequest request,
+ JsonTypeInfo jsonTypeInfo,
+ CancellationToken cancellationToken = default);

+ public static IAsyncEnumerable ReadFromJsonLinesAsAsyncEnumerable(
+ this HttpRequest request,
+ JsonTypeInfo jsonTypeInfo,
+ CancellationToken cancellationToken = default);
}

public static class HttpResponseJsonExtensions
{
+ public static Task WriteAsJsonLinesAsync(
+ this HttpResponse response,
+ IAsyncEnumerable values,
+ JsonSerializerOptions? options = null,
+ string? contentType = null,
+ CancellationToken cancellationToken = default);

+ public static Task WriteAsJsonLinesAsync(
+ this HttpResponse response,
+ IAsyncEnumerable values,
+ JsonTypeInfo jsonTypeInfo,
+ string? contentType = null,
+ CancellationToken cancellationToken = default);
}

namespace Microsoft.AspNetCore.Http;

public static class Results
{
+ public static IResult JsonLines(
+ IAsyncEnumerable values,
+ JsonSerializerOptions? options = null,
+ string? contentType = null);
+
+ public static IResult JsonLines(
+ IAsyncEnumerable values,
+ JsonTypeInfo jsonTypeInfo,
+ string? contentType = null);
}

namespace Microsoft.AspNetCore.Builder;

public static class OpenApiRouteHandlerBuilderExtensions
{
+ public static TBuilder ProducesJsonLines(
+ this TBuilder builder,
+ int statusCode = StatusCodes.Status200OK,
+ string? contentType = null)
+ where TBuilder : IEndpointConventionBuilder;
}
```

Behavioral notes:

- The default content type is `application/jsonl; charset=utf-8`.
- Each element in `IAsyncEnumerable` is serialized as a complete JSON value followed by `\n`.
- The request APIs return an `IAsyncEnumerable` so callers can process values incrementally without requiring the entire request body to be buffered.
- JSON Lines support is explicit; existing `ReadFromJsonAsync` and `WriteAsJsonAsync` APIs keep their current single-document JSON semantics.
- MVC integration is expected to be provided by built-in JSON Lines formatters registered alongside existing JSON formatters, but those formatter types are not proposed as public API.

## Usage Examples

### Minimal API response streaming

```csharp
app.MapGet("/events", () =>
{
return Results.JsonLines(GetEventsAsync());
});

static async IAsyncEnumerable GetEventsAsync()
{
yield return new WeatherEvent("start", DateTimeOffset.UtcNow);
await Task.Delay(100);
yield return new WeatherEvent("update", DateTimeOffset.UtcNow);
await Task.Delay(100);
yield return new WeatherEvent("complete", DateTimeOffset.UtcNow);
}

public sealed record WeatherEvent(string Type, DateTimeOffset Timestamp);
```

Example response body:

```text
{"type":"start","timestamp":"2026-06-14T10:00:00+00:00"}
{"type":"update","timestamp":"2026-06-14T10:00:00.1+00:00"}
{"type":"complete","timestamp":"2026-06-14T10:00:00.2+00:00"}
```

### Minimal API request streaming

```csharp
app.MapPost("/ingest", async (HttpRequest request, CancellationToken cancellationToken) =>
{
await foreach (var item in request.ReadFromJsonLinesAsAsyncEnumerable(cancellationToken: cancellationToken))
{
if (item is null)
{
continue;
}

await ProcessAsync(item, cancellationToken);
}

return Results.Accepted();
});

public sealed record IngestItem(string Id, string Payload);
```

### Source-generated metadata for AOT-friendly scenarios

```csharp
app.MapGet("/todos", (HttpContext context, CancellationToken cancellationToken) =>
{
return context.Response.WriteAsJsonLinesAsync(
GetTodosAsync(),
TodoJsonContext.Default.Todo,
cancellationToken: cancellationToken);
});

[JsonSerializable(typeof(Todo))]
public partial class TodoJsonContext : JsonSerializerContext
{
}
```

### OpenAPI / endpoint metadata

```csharp
app.MapGet("/logs", () => TypedResults.JsonLines(GetLogEntriesAsync()))
.ProducesJsonLines(StatusCodes.Status200OK);
```

### MVC controller action

```csharp
[ApiController]
[Route("[controller]")]
public sealed class FeedController : ControllerBase
{
[HttpGet]
[Produces("application/jsonl")]
public IAsyncEnumerable Get(CancellationToken cancellationToken)
=> GetFeedAsync(cancellationToken);

private static async IAsyncEnumerable GetFeedAsync(
[EnumeratorCancellation] CancellationToken cancellationToken)
{
yield return new FeedItem("1", "first");
await Task.Delay(10, cancellationToken);
yield return new FeedItem("2", "second");
}
}

public sealed record FeedItem(string Id, string Message);
```

## Alternative Designs

### Reuse `WriteAsJsonAsync` and `ReadFromJsonAsync`

One option is to extend the existing JSON APIs so that:
- `WriteAsJsonAsync(IAsyncEnumerable)` could emit JSON Lines when the content type is `application/jsonl`
- `ReadFromJsonAsync>()` could read JSON Lines for `application/jsonl`

This was not selected because it overloads existing APIs with fundamentally different framing semantics. Today, `WriteAsJsonAsync` means “write one JSON document”. Changing that meaning based on type or content type would make behavior less predictable and harder to discover.

### Return `Task>` or `Task` for request-side APIs

Another option is to add APIs that read a JSON Lines request body into a buffered collection.

This was not selected as the primary shape because it gives up the main benefit of JSON Lines: incremental processing. It also invites confusion with ordinary JSON arrays. Buffered helpers could be added later if there is demand.

### Add public MVC formatter types as the API surface

Another option is to expose public formatter types such as `SystemTextJsonLinesInputFormatter` and `SystemTextJsonLinesOutputFormatter`.

This was not selected because:
- most app code benefits more from high-level HTTP APIs than formatter types
- formatter registration can remain an implementation detail
- the repo already exposes high-level JSON request/response APIs that are a better fit for minimal APIs and general HTTP usage

### Support synchronous `IEnumerable` in the first version

It would be possible for response APIs to also accept `IEnumerable`.

This was not selected for the initial proposal because `IAsyncEnumerable` is the strongest match for network streaming and backpressure-aware production. Supporting `IEnumerable` can be considered later if needed.

### Add implicit JSON Lines binding for all `application/jsonl` requests

Another option is to automatically bind `application/jsonl` request bodies to collection types such as `List` or arrays in MVC and minimal APIs.

This was not selected initially because it is less clear how partial failures, streaming, cancellation, and buffering should behave. Starting with explicit streaming APIs provides clearer semantics.

## Risks

- **Runtime dependency**: this proposal assumes foundational JSON Lines serializer/deserializer support in `System.Text.Json`, as tracked by `dotnet/runtime#126395`. The final ASP.NET Core API shape may need to align with whatever runtime surface is approved.
- **Error semantics for partial reads**: when reading request bodies as an async stream, failures may occur after some items were already consumed. This differs from the all-or-nothing behavior of deserializing a single JSON document and should be carefully documented.
- **Content negotiation complexity**: JSON Lines should not silently replace ordinary JSON array output for existing endpoints. The implementation must keep `application/json` and `application/jsonl` semantics distinct.
- **OpenAPI/tooling support**: some OpenAPI tooling may not model streaming newline-delimited payloads especially well, even when the media type is correctly described.
- **Interoperability naming**: some ecosystems use `application/x-ndjson`. If interoperability feedback shows that alias is important, the implementation may need to support both media types while still documenting a single preferred one.
- **Cancellation and flushing behavior**: response streaming must balance throughput, buffering, and latency. Flushing every line may be useful for real-time scenarios but could reduce throughput in high-volume cases if not implemented carefully.

Contributor guide

Open the contributing guide

Research direction

Start with dotnet/runtime#126395, then compare the proposed HttpRequestJsonExtensions, HttpResponseJsonExtensions, Results.JsonLines, and ProducesJsonLines entry points with the existing JSON APIs named in the issue. Review how minimal APIs, MVC formatters, content negotiation, endpoint metadata, cancellation, and partial-read errors fit together. Done means explicit JSON Lines request and response support, streaming IAsyncEnumerable behavior, and the documented application/jsonl semantics described here.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.