Azure / Azure/azure-sdk

SearchClient throws JsonReaderException while ingesting data and Azure Cognitive Search is throttling

Open
#8,495 0 comments 0 reactions 0 assignees View on GitHub
Client customer-reported needs-team-triage question Search
Dominant language
PowerShell
Stars
597
Forks
374
Avg merge
5d 21h
Merged PRs (30d)
42

Description

I want to ingest about 15million documents to Azure Cognitive search (S0) with a batch size of 5000.

After reaching around 70% the `SearchClient` is throwing the following exception

> System.Text.Json.JsonReaderException: The input does not contain any JSON tokens. Expected the input to start with a valid JSON token, when isFinalBlock is true. LineNumber: 0 | BytePositionInLine: 0.
> at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
> at System.Text.Json.JsonDocument.Parse(ReadOnlySpan`1 utf8JsonSpan, JsonReaderOptions readerOptions, MetadataDb& database, StackRowStack& stack)
> at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory`1 utf8Json, JsonReaderOptions readerOptions, Byte[] extraRentedArrayPoolBytes, PooledByteBufferWriter extraPooledByteBufferWriter)
> at System.Text.Json.JsonDocument.ParseAsyncCore(Stream utf8Json, JsonDocumentOptions options, CancellationToken cancellationToken)
> at Azure.Search.Documents.SearchClient.IndexDocumentsInternal[T](IndexDocumentsBatch`1 batch, IndexDocumentsOptions options, Boolean async, CancellationToken cancellationToken)
> at Azure.Search.Documents.SearchClient.IndexDocumentsAsync[T](IndexDocumentsBatch`1 batch, IndexDocumentsOptions options, CancellationToken cancellationToken)
> at Azure.Search.Documents.SearchClient.MergeOrUploadDocumentsAsync[T](IEnumerable`1 documents, IndexDocumentsOptions options, CancellationToken cancellationToken)
>

During this time I can see in the logs that Azure Search is throttling/denying requests with status 429 (Too Many Requests).
Library: Azure.Search.Documents 11.5.1

My guess is that the library has an issue parsing the response correctly.

Contributor guide

Open the contributing guide

Research direction

Start with Azure.Search.Documents 11.5.1 and the SearchClient.IndexDocumentsInternal path named in the stack trace. Reproduce ingestion under Azure Search 429 throttling and inspect how the response is parsed when it contains no JSON tokens. Done means throttled requests produce a handled, actionable result instead of JsonReaderException.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
api, backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.