[API Proposal]: Add incremental chunked reading support to CborReader similar to Utf8JsonReader style
- Dominant language
- C#
- Stars
- 18.3k
- Forks
- 5.6k
- PR merge metrics
- PR metrics pending
Description
### Background and motivation
`CborReader` uses single buffer and when a buffer ends mid item `PeekState()` and `Read*` methods throw `CborContentException `, and `Reset(ReadOnlyMemory)` the only API to update the buffer discards all nesting state.
So consumers reading CBOR from a Stream in fixed chunks must either load the whole stream or drive control flow with try/catch.
`Utf8JsonReader` on the other hand fixes this gap by adding `isFinalBlock` and `CurrentState`. CBOR should have similar synchronous incremental mode, with the caller owning I/O (sync or async).
### API Proposal
```csharp
namespace System.Formats.Cbor;
public partial enum CborReaderState
{
NeedsMoreData,
}
public partial class CborReader
{
// Existing:
// public CborReader(ReadOnlyMemory data, CborReaderOptions? options);
// So isFinalBlock could be defaulted, or not.
//
// throws ArgumentException if isFinalBlock is false and options.ConformanceMode != Lax.
public CborReader(ReadOnlyMemory data, CborReaderOptions? options, bool isFinalBlock);
// Replaces the current _data and sets the instance _offset to 0, but does not reset the
// array/map nesting stack.
//
// throws InvalidOperationException if _isFinalBlock is true (cannot Slide once the end is known)
public void SlideData(ReadOnlyMemory data, bool isFinalBlock);
// Replaces the current _data and sets the instance _offset to 0, and resets
// array/map nesting stack.
//
// Existing: public void Reset(ReadOnlyMemory data);
public void Reset(ReadOnlyMemory data, bool isFinalBlock);
// true: All cases where SkipValue would not throw. _offset has moved to the end of the value.
// false: SkipValue would have thrown "unexpected end of buffer" and _isFinalBlock is false
// throws: All other reasons (e.g. malformed input)
//
// Note: Until/unless a solution is landed on to enable isFinalBlock for !Lax,
// the boolean has no measurable effect. But it matches the signature of SkipValue.
public bool TrySkipValue(bool disableConformanceModeChecks = false);
public bool TrySkipToParent(bool disableConformanceModeChecks = false);
}
```
* ctor or ResetData with isFinalBlock:true preserve today's behavior for all methods
* The new SlideData method will throw InvalidOperationException
* isFinalBlock:false
* Ctor throws unless conformance mode is Lax (v1 restriction)
* Read\* still succeeds or throws as today
* SkipValue and SkipToParent still succeed or throw as today
* PeekState reports NeedsMoreData when the object at the current offset is incomplete, rather than throwing CborContentException. Still throws in other cases.
* TrySkipValue/TrySkipToParent return true when SkipValue/SkipToParent would succeed, false when they would fail due to data truncation, throw when data is invalid (e.g. an indefinite map with an odd number of entries, violating key/value parity)
* SlideData resets _data to the new value and _offset to 0, but remembers CurrentDepth and the state push/pop stack.
* Reset resets _data to the new value, _offset to 0, and clears the state push/pop stack (resetting CurrentDepth to 0).
### API Usage
```csharp
static async Task ReadCborFromStreamAsync(Stream stream, CancellationToken ct)
{
// Two-arg Reset is supported only in Lax conformance mode.
var reader = new CborReader(ReadOnlyMemory.Empty, CborConformanceMode.Lax);
byte[] buffer = new byte[4096];
int dataLength = await stream.ReadAsync(buffer, ct);
bool isFinalBlock = dataLength == 0;
reader.Reset(buffer.AsMemory(0, dataLength), isFinalBlock);
MyValue result = default;
while (true)
{
switch (reader.PeekState())
{
case CborReaderState.NeedsMoreData:
// In Lax the unconsumed tail IS the leftover, so BytesRemaining is all we need.
int keep = reader.BytesRemaining;
int start = dataLength - keep;
if (keep == buffer.Length) // one item bigger than the buffer
Array.Resize(ref buffer, buffer.Length * 2);
Buffer.BlockCopy(buffer, start, buffer, 0, keep);
int read = await stream.ReadAsync(buffer.AsMemory(keep), ct);
dataLength = keep + read;
isFinalBlock = read == 0;
reader.SlideData(buffer.AsMemory(0, dataLength), isFinalBlock); // nesting preserved
continue;
case CborReaderState.Finished:
return result; // whole document consumed
case CborReaderState.*: //One token read is safe as `PeekState() == NeedsMoreData` checked before as the non-throwing gate
result = reader.Read*();
break;
// ... other typed cases ...
default:
throw new InvalidOperationException($"Unexpected state {reader.PeekState()}.");
}
}
}
```
### Alternative Designs
Stream/async overloads on CborReader (the rejected #99993 ask). Implies async variants of every Read* (the maintainer objection). The isFinalBlock model keeps the reader synchronous and the caller owns I/O.
External JsonReaderState-style state struct + new-reader-per-chunk. Exists only because `Utf8JsonReader` is a ref struct however CborReader is a class, so in-place preservation via the Reset overload is simpler.
Changing TryReadByteString/TryReadTextString to also return false on needs-more-data: overloads false with two incompatible recoveries (grow output vs refill input) the caller can't disambiguate, and is redundant because the PeekState() gate already guarantees a definite string is buffered before the read. TrySkipValue is the sole Try* that signals needs-more-data, justified because no single-token peek can gate a multi-token subtree skip and its false is unambiguous.
### Risks
Additive enum member. `CborReaderState.NeedsMoreData` can appear in a switch with { default: throw } but it is returned only when the caller opts into non-final mode, so existing single-buffer callers shouldn't observe it.
`PeekState()` behavior change is scoped to non-final mode. Final mode (and the one-arg Reset) is byte-for-byte unchanged.
Buffer-size floor. Non-final mode can't progress if the buffer is smaller than the largest single definite string value. Same as the documented `Utf8JsonReader` constraint.
`TrySkipValue` buffers the skipped subtree (rewind-and-retry); fine for small unknown fields, resumable streaming skip is can be added as a follow up if needed.
`NotSupportedException` for non-Lax is a deliberate scope limit and doesn't have to be permanent.
Contributor guide
Assessment
This issue has not been assessed yet.