dotnet / dotnet/runtime

[API Proposal]: Add incremental chunked reading support to CborReader similar to Utf8JsonReader style

Open
#129,718 20 comments 1 reaction 1 assignee Claimed by @muhammad-othman View on GitHub
api-approved area-System.Formats.Cbor
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

### Background and motivation

`CborReader` uses single buffer and when a buffer ends mid item `PeekState()` and `Read*` methods throw `CborContentException `, and `Reset(ReadOnlyMemory)` the only API to update the buffer discards all nesting state.
So consumers reading CBOR from a Stream in fixed chunks must either load the whole stream or drive control flow with try/catch.
`Utf8JsonReader` on the other hand fixes this gap by adding `isFinalBlock` and `CurrentState`. CBOR should have similar synchronous incremental mode, with the caller owning I/O (sync or async).

### API Proposal

```csharp
namespace System.Formats.Cbor;

public partial enum CborReaderState
{
NeedsMoreData,
}

public partial class CborReader
{
// Existing:
// public CborReader(ReadOnlyMemory data, CborReaderOptions? options);
// So isFinalBlock could be defaulted, or not.
//
// throws ArgumentException if isFinalBlock is false and options.ConformanceMode != Lax.
public CborReader(ReadOnlyMemory data, CborReaderOptions? options, bool isFinalBlock);

// Replaces the current _data and sets the instance _offset to 0, but does not reset the
// array/map nesting stack.
//
// throws InvalidOperationException if _isFinalBlock is true (cannot Slide once the end is known)
public void SlideData(ReadOnlyMemory data, bool isFinalBlock);

// Replaces the current _data and sets the instance _offset to 0, and resets
// array/map nesting stack.
//
// Existing: public void Reset(ReadOnlyMemory data);
public void Reset(ReadOnlyMemory data, bool isFinalBlock);

// true: All cases where SkipValue would not throw. _offset has moved to the end of the value.
// false: SkipValue would have thrown "unexpected end of buffer" and _isFinalBlock is false
// throws: All other reasons (e.g. malformed input)
//
// Note: Until/unless a solution is landed on to enable isFinalBlock for !Lax,
// the boolean has no measurable effect. But it matches the signature of SkipValue.
public bool TrySkipValue(bool disableConformanceModeChecks = false);
public bool TrySkipToParent(bool disableConformanceModeChecks = false);
}
```

* ctor or ResetData with isFinalBlock:true preserve today's behavior for all methods
* The new SlideData method will throw InvalidOperationException
* isFinalBlock:false
* Ctor throws unless conformance mode is Lax (v1 restriction)
* Read\* still succeeds or throws as today
* SkipValue and SkipToParent still succeed or throw as today
* PeekState reports NeedsMoreData when the object at the current offset is incomplete, rather than throwing CborContentException. Still throws in other cases.
* TrySkipValue/TrySkipToParent return true when SkipValue/SkipToParent would succeed, false when they would fail due to data truncation, throw when data is invalid (e.g. an indefinite map with an odd number of entries, violating key/value parity)
* SlideData resets _data to the new value and _offset to 0, but remembers CurrentDepth and the state push/pop stack.
* Reset resets _data to the new value, _offset to 0, and clears the state push/pop stack (resetting CurrentDepth to 0).

### API Usage

```csharp

static async Task ReadCborFromStreamAsync(Stream stream, CancellationToken ct)
{
// Two-arg Reset is supported only in Lax conformance mode.
var reader = new CborReader(ReadOnlyMemory.Empty, CborConformanceMode.Lax);

byte[] buffer = new byte[4096];
int dataLength = await stream.ReadAsync(buffer, ct);
bool isFinalBlock = dataLength == 0;
reader.Reset(buffer.AsMemory(0, dataLength), isFinalBlock);
MyValue result = default;

while (true)
{
switch (reader.PeekState())
{
case CborReaderState.NeedsMoreData:
// In Lax the unconsumed tail IS the leftover, so BytesRemaining is all we need.
int keep = reader.BytesRemaining;
int start = dataLength - keep;

if (keep == buffer.Length) // one item bigger than the buffer
Array.Resize(ref buffer, buffer.Length * 2);

Buffer.BlockCopy(buffer, start, buffer, 0, keep);
int read = await stream.ReadAsync(buffer.AsMemory(keep), ct);
dataLength = keep + read;
isFinalBlock = read == 0;

reader.SlideData(buffer.AsMemory(0, dataLength), isFinalBlock); // nesting preserved
continue;

case CborReaderState.Finished:
return result; // whole document consumed

case CborReaderState.*: //One token read is safe as `PeekState() == NeedsMoreData` checked before as the non-throwing gate
result = reader.Read*();
break;

// ... other typed cases ...

default:
throw new InvalidOperationException($"Unexpected state {reader.PeekState()}.");
}
}
}

```

### Alternative Designs

Stream/async overloads on CborReader (the rejected #99993 ask). Implies async variants of every Read* (the maintainer objection). The isFinalBlock model keeps the reader synchronous and the caller owns I/O.

External JsonReaderState-style state struct + new-reader-per-chunk. Exists only because `Utf8JsonReader` is a ref struct however CborReader is a class, so in-place preservation via the Reset overload is simpler.

Changing TryReadByteString/TryReadTextString to also return false on needs-more-data: overloads false with two incompatible recoveries (grow output vs refill input) the caller can't disambiguate, and is redundant because the PeekState() gate already guarantees a definite string is buffered before the read. TrySkipValue is the sole Try* that signals needs-more-data, justified because no single-token peek can gate a multi-token subtree skip and its false is unambiguous.

### Risks

Additive enum member. `CborReaderState.NeedsMoreData` can appear in a switch with { default: throw } but it is returned only when the caller opts into non-final mode, so existing single-buffer callers shouldn't observe it.
`PeekState()` behavior change is scoped to non-final mode. Final mode (and the one-arg Reset) is byte-for-byte unchanged.
Buffer-size floor. Non-final mode can't progress if the buffer is smaller than the largest single definite string value. Same as the documented `Utf8JsonReader` constraint.
`TrySkipValue` buffers the skipped subtree (rewind-and-retry); fine for small unknown fields, resumable streaming skip is can be added as a follow up if needed.
`NotSupportedException` for non-Lax is a deliberate scope limit and doesn't have to be permanent.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.