CommunityToolkit / CommunityToolkit/AI

Add VisionLMOcrClient: an IOcrClient backed by a vision-capable IChatClient

Open
#13 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
7
Forks
4
Avg merge
5h 54m
Merged PRs (30d)
8

Description

## Summary

Add a **`VisionLMOcrClient`** — a provider-neutral `IOcrClient` implementation backed by any
`Microsoft.Extensions.AI.IChatClient` that supports image input (a vision LM). It turns "give me an
`IChatClient` that can see" into a first-class OCR provider that any MEDI reader (e.g. the PdfPig
reader) can compose, alongside dedicated OCR services (Mistral Document AI, Azure Document
Intelligence).

This is the vision-LM path from the current PdfPig `VisionOcrEnricher`, **relocated and reshaped**:
instead of a downstream document *processor* that mutates placeholder elements, it becomes an
`IOcrClient` provider the reader calls inline. Same mechanism (prompt + page image → text), correct
layer.

## Why (motivation)

- A vision LM *is* an OCR engine — but today the only way to use one for OCR in MEDI is the PdfPig
`VisionOcrEnricher`, which is coupled to the reader's placeholder/`page_image` stash and lives at
the wrong layer (post-read processor, not an OCR provider).
- The `IOcrClient` abstraction (dotnet/extensions #7588) is the provider-neutral seam for
"document → structured text". A vision LM should be *one provider behind that seam*, swappable
with Mistral OCR / Azure DI at composition time — no reader changes.
- Keeping the concrete impl in CommunityToolkit (not dotnet/extensions) respects the layering rule:
**extensions holds only the abstraction; concrete providers live in the toolkit, none
privileged.**

## Package

Proposed: **`CommunityToolkit.DataIngestion.VisionLMOcr`** (name open to discussion — the point is a
*satellite* package whose only extra dependency footprint is `IChatClient`; it does not force a
vision dependency onto the base ingestion package).

Dependencies: `Microsoft.Extensions.AI.Abstractions` + the `IOcrClient`/`OcrResult` abstractions
only. **No** provider SDKs. Neutral by construction.

Targets: `net10.0;net8.0` (match sibling packages).

## The `IOcrClient` contract to implement

> **Note:** `IOcrClient` and its result types are currently **unpublished** — they live in
> dotnet/extensions PR **#7588**. Until that merges, **vendor** the shape below into this package
> (internal copy). When #7588 ships, delete the vendored copy and reference the real
> `Microsoft.Extensions.DataIngestion.Abstractions` types. This is a tracked, deliberate temporary
> duplication.

```csharp
public interface IOcrClient
{
Task GetTextAsync(
Stream document,
string mediaType,
OcrOptions? options = null,
IProgress? progress = null,
CancellationToken cancellationToken = default);

object? GetService(Type serviceType, object? serviceKey = null);
}

public sealed class OcrResult(IReadOnlyList pages)
{
public IReadOnlyList Pages { get; } = pages;
public string Markdown => string.Join("\n\n", Pages.Select(p => p.Markdown));
public string? OcrSource { get; init; }
public string? ModelId { get; init; }
public OcrUsage? Usage { get; init; }
// + RawRepresentation, AdditionalProperties
}

public sealed class OcrPage(int index, string markdown)
{
public int Index { get; }
public string Markdown { get; }
public IReadOnlyList Blocks { get; init; } = [];
public IReadOnlyList Tables { get; init; } = [];
public double? Confidence { get; init; }
}
// OcrTable { RowCount, ColumnCount, OcrTableCell[,]? Cells, string MarkdownRepresentation, ToMarkdown() }
// OcrBlock { Text, Kind?, OcrBoundingRegion?, Confidence? }
// OcrBoundingRegion { PageNumber, Polygon, FromRectangle(...), GetBounds() }
// OcrOptions { ModelId?, IncludeImages, AdditionalProperties? }
// OcrProgress { PagesProcessed?, TotalPages?, Status? }
// OcrUsage { PagesProcessed?, AdditionalProperties? }
```

## Behavior spec

`VisionLMOcrClient(IChatClient chatClient, VisionLMOcrOptions? options = null) : IOcrClient`

1. **Input.** `GetTextAsync(document, mediaType, ...)` receives a document stream + media type.
- If `mediaType` is an image (`image/png`, `image/jpeg`, …): send the bytes directly as one page.
- If it's a rendered-page image handed in by a reader (the common PdfPig `FallbackForEmptyPages`
case): one image → one `OcrPage`.
- (PDF rasterization is **out of scope** — the reader renders pages to images and calls this per
page. This client OCRs *images*.)
2. **Prompt.** Reuse the proven `VisionOcrEnricher` prompt:
- System: *"You are a precise OCR engine. Extract all visible text from the provided image
exactly as it appears. Preserve line breaks and formatting. Output only the extracted text, no
commentary."*
- User: `DataContent(imageBytes, mediaType)` + `TextContent("Extract all text from this image.")`
3. **Call** `chatClient.GetResponseAsync(messages, ct)`; map `response.Text` → `OcrPage.Markdown`.
4. **Result.** Build `OcrResult` with `OcrSource = "vision_lm"`, `ModelId` from options/response.
Populate `OcrPage.Tables` **only if** the model returns a table representation (optional; leave
empty otherwise — do **not** invent a separate table pass; tables at the OCR layer come from the
provider, and a chat LM generally returns them inline in markdown).
5. **Progress.** Emit `OcrProgress` per page when `IProgress` is supplied.
6. **`GetService`.** Return `this` for `IOcrClient`/`VisionLMOcrClient`; delegate to the inner
`IChatClient.GetService` otherwise.

### Options

```csharp
public sealed class VisionLMOcrOptions
{
public string? ModelId { get; set; }
public string? SystemPrompt { get; set; } // default = the OCR prompt above
public string? UserPrompt { get; set; } // default = "Extract all text from this image."
}
```

### DI extension

```csharp
services.AddVisionLMOcrClient(); // resolves IChatClient from DI
services.AddVisionLMOcrClient(o => o.ModelId = "gpt-4o");
```

Register as `IOcrClient` so readers pick it up by composition.

## Tests (unit, no network)

Use a **fake `IChatClient`** (returns canned `ChatResponse`):

- Image input → one `OcrPage` whose `Markdown` == the fake's returned text; `OcrSource == "vision_lm"`.
- The fake receives a `DataContent` with the right media type + the OCR system prompt.
- Custom `SystemPrompt`/`UserPrompt`/`ModelId` from options are threaded through.
- `IProgress` fires once per page.
- `GetService(typeof(IOcrClient))` returns the client; unknown types delegate to inner chat client.
- Build + tests green on `net8.0` and `net10.0`.

## Acceptance criteria

- [ ] New package `CommunityToolkit.DataIngestion.VisionLMOcr` (or agreed name), `net10.0;net8.0`.
- [ ] `VisionLMOcrClient : IOcrClient` over `IChatClient`, prompt/behavior per spec.
- [ ] `VisionLMOcrOptions` + `AddVisionLMOcrClient(...)` DI extension.
- [ ] Vendored `IOcrClient` shape (internal) with a clear TODO to un-vendor when #7588 merges.
- [ ] Unit tests above, green net8/net10.
- [ ] No provider SDK dependencies — `IChatClient` + OCR abstractions only.

## Dependency / sequencing note

- **Gated on dotnet/extensions #7588** for the *final* form (real `IOcrClient`). Ship now with the
vendored shape; swap when #7588 lands.
- Sibling PR replatforms the PdfPig reader onto `IOcrClient` (`PdfPigOcrReader` + `OcrPolicy`) and
removes the old `VisionOcrEnricher`; this client is that vision path's new home.

## Reference: the logic being relocated

The current `VisionOcrEnricher` (PdfPig reader PR) already implements the vision-LM OCR call — prompt,
`DataContent` image message, `GetResponseAsync`, text extraction. This issue **moves that mechanism**
behind `IOcrClient` and drops the placeholder/`page_image`-stash coupling. A junior dev can port the
message-construction + response-mapping code almost verbatim.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.