Chat template defaulting to GitHub Models means ingestion is impractical beyond 'hello world' case
- Dominant language
- C#
- Stars
- 3.2k
- Forks
- 894
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 23
Description
I was trying out ingesting data beyond the two sample PDFs and quickly ran into rate limit issues:
```
Unhandled exception. System.ClientModel.ClientResultException: HTTP 429 (: RateLimitReached)
Rate limit of 20 per 60s exceeded for UserByModelByMinute. Please wait 0 seconds before retrying.
at OpenAI.ClientPipelineExtensions.ProcessMessageAsync(ClientPipeline pipeline, PipelineMessage message, RequestOptions options)
```
This rate limit is unfortunate. More info about it:

This leads to several problems with the existing ingestion mechanism:
1. It runs out of requests very quickly. Currently it calls the generator once per source document, which means you couldn't ingest more than 150 docs/day. In any case, you'd hit per-minute rate limits much sooner than that (e.g., eShopSupport tries to ingest 200 docs, and would hit per-minute limits every 15 docs)
2. We don't handle rate limit errors. They just cause ingestion to fail.
3. We don't manage the number of input tokens. We just send all the chunks in an entire source document, even if the total is over 64k tokens, which would also fail.
What we have is OK for "hello world" cases (2 small PDFs) but doesn't address the question of scaling beyond that. It's good that the template is labelled "preview" because this is quite a basic issue we need to resolve.
Possible strategies:
* We could do in-proc embedding generation (e.g., via Onnx runtime) instead of calling an external embedding generator service
* We could reconsider the whole approach around ingestion to make it less frameworky and more clearly just one special case for a "getting started" template, making it more obvious that developers are responsible for figuring out their own approach to ingestion based on their needs
* ... or we could go the other way and actually try to make something like a minimal ingestion framework that handles all the edge cases
Contributor guide
Assessment
This issue has not been assessed yet.