dotnet / dotnet/extensions

[MEDI] SemanticSimilarityChunker raising OllamaSharp.Models.Exceptions.OllamaException: the input length exceeds the context length

Open
#7,297 10 comments 0 reactions 1 assignee Claimed by @adamsitnik View on GitHub
area-data-ingestion bug
Dominant language
C#
Stars
3.2k
Forks
894
Avg merge
1d 12h
Merged PRs (30d)
23

Description

### Description

Hi.

Probably this is caused by some misunderstanding mine. If it is please explain it to me.
I'm using the AI chat template as it is. The only changes: I'm setting the MaxTokensPerChunk and I'm using ElBruno.OllamaSharp.Extensions to extend the timeout.

```
IChatClient chatClient = new OllamaApiClient(new Uri("http://localhost:11434"),
"llama3.2").WithLongTimeout();
IEmbeddingGenerator> embeddingGenerator = new OllamaApiClient( new Uri("http://localhost:11434"),
"all-minilm").WithLongTimeout();
```

VectorStore: SqliteVectorStore

Note: I tried to use a PostgreSQLVectorStore but results are worst, even bringing MaxTokensPerChunk to 100 or less raises the exception.

In DataIngestor.IngestDataAsync I have the following:

``` lang:csharp
using var pipeline = new IngestionPipeline(
reader: new DocumentReader(directory),
chunker: new SemanticSimilarityChunker(embeddingGenerator,
new IngestionChunkerOptions(TiktokenTokenizer.CreateForModel("gpt-4o")){
MaxTokensPerChunk = 200 }
),
writer: writer,
loggerFactory: loggerFactory);

await foreach (var result in pipeline.ProcessAsync(directory, searchPattern))
{
logger.LogInformation("Completed processing '{id}'. Succeeded: '{succeeded}'.", result.DocumentId, result.Succeeded);
}
```

I have 3 pdf files to be ingested. 2 of them are perfectly ingested. 1 is raising the exception listed below.

If I change to another checker like DocumentTokenChunker, all 3 are ingested.

Is SemanticSimilarityChunker ignoring the MaxTokensPerChunk?
Am I doing anything wrong?

```
Microsoft.Extensions.DataIngestion.IngestionPipeline[6]
An error occurred while ingesting document 'my.pdf'.
OllamaSharp.Models.Exceptions.OllamaException: the input length exceeds the context length
at OllamaSharp.OllamaApiClient.EnsureSuccessStatusCodeAsync(HttpResponseMessage response)
at OllamaSharp.OllamaApiClient.SendToOllamaAsync(HttpRequestMessage requestMessage, OllamaRequest ollamaRequest, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at OllamaSharp.OllamaApiClient.PostAsync[TRequest,TResponse](String endpoint, TRequest ollamaRequest, CancellationToken cancellationToken)
at OllamaSharp.OllamaApiClient.Microsoft.Extensions.AI.IEmbeddingGenerator>.GenerateAsync(IEnumerable`1 values, EmbeddingGenerationOptions options, CancellationToken cancellationToken)
at Microsoft.Extensions.DataIngestion.Chunkers.SemanticSimilarityChunker.CalculateDistancesAsync(IngestionDocument documents, CancellationToken cancellationToken)
at Microsoft.Extensions.DataIngestion.Chunkers.SemanticSimilarityChunker.ProcessAsync(IngestionDocument document, CancellationToken cancellationToken)+MoveNext()
at Microsoft.Extensions.DataIngestion.Chunkers.SemanticSimilarityChunker.ProcessAsync(IngestionDocument document, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource.GetResult()
at Microsoft.Extensions.DataIngestion.VectorStoreWriter`1.WriteAsync(IAsyncEnumerable`1 chunks, CancellationToken cancellationToken)
at Microsoft.Extensions.DataIngestion.VectorStoreWriter`1.WriteAsync(IAsyncEnumerable`1 chunks, CancellationToken cancellationToken)
at Microsoft.Extensions.DataIngestion.IngestionPipeline`1.IngestAsync(IngestionDocument document, Activity parentActivity, CancellationToken cancellationToken)
at Microsoft.Extensions.DataIngestion.IngestionPipeline`1.ProcessAsync(IEnumerable`1 files, Activity rootActivity, CancellationToken cancellationToken)+MoveNext()

```

### Reproduction Steps

Use a relative sized PDF file
use SemanticSimilarityChunker
use local ollama from the AI Chat template wizard
use Sqlite Vector Store

### Expected behavior

All 'big' PDFs should be ingested.

### Actual behavior

The biggest among those is not ingested

### Regression?

_No response_

### Known Workarounds

_No response_

### Configuration

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.