Generalize chat template's approach to citations
- Dominant language
- C#
- Stars
- 3.2k
- Forks
- 894
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 23
Description
In the initial preview, citations are very limited:
* Search results have to be of the form `(filename, page_number, text)`. This only makes sense for the default scenario of PDFs in a directory. Changing it to represent other citation sources is difficult and nonobvious.
* The code that displays links to view citations does so by checking `File.EndsWith(".pdf")`. We haven't made it very helpful if you need to show UI for other types of citation.
Proposal: we should generalize this so it's more obvious how you can plug in other retrieval sources. For example if you want the RAG retrieval phase to use an existing semantic search API on some external CMS, you want the citations to use arbitrary URLs (not filename/pagenumber pairs).
### Underlying details
For the first preview, one of the simplifying assumptions we made is that citations can be round-tripped through the LLM without being stored anywhere else. This makes the code easier but imposes limtations.
That is, today, we expect the "search" `AIFunction` to return results like:
```xml
This is text from somefile1
This is highly relevant information
```
... and we instruct the LLM to return output including markup like:
```xml
exact quote here
```
So, there's no other storage for the search results. We just pass them into the LLM and accept whatever it gives us back. This is limiting because if you wanted to store other arbitrary and potentially large metadata for each result, it would have to pass through the LLM and then:
* The fact that it's large is costing you input tokens
* The fact that it's large means the LLM might not reflect it back with 100% fidelity. For example if it was a massive URL including a GUID or some base64-encoded info, you shouldn't assume the LLM will repeat it back verbatim. It might change some characters in the GUID/base64 or hallucinate other querystring parameters.
### A more general solution
We should change this to work more like it does in eShopSupport:
* We should define a class/record to represent "citation". Initially this would have properties `(string Type, string? Filename, int? PageNumber)` but developers would be free to add other properties specific to their data sources (e.g., `Url`) without having to change the LLM prompt or citation parsing code.
* The "search" `AIFunction`s should:
1. Populate a `ConcurrentDictionary` (scoped to the `Chat.razor` component instance) with each search result, where the key is some incrementing ID (also scoped to the component instance).
2. Return results like `This is text from somefile1This is highly relevant information`
* The LLM should be instructed to give citations like `short quote here`
* The UI should use the ID to look up the dictionary entry, and hence recover its full details including arbitrarily-long URLs or any other data the app developer has associated with each result
* The UI for displaying citations will look for `type == "pdf"`, and hence it will be relatively obvious that developers can add further `.razor` logic to display other UI for other values of `type` and use other custom properties like `Url`.
Although this would add a bit more code to the template, hopefully it would not be a lot of code, and it would clarify a lot about how to go beyond the basic "PDFs in a local directory" situation.
Contributor guide
Assessment
This issue has not been assessed yet.