aiondemand / aiondemand/aiod-enhanced-interaction
Utilize third-party API endpoints for invoking LLM calls
- Vorherrschende Sprache
- Python
- Sterne
- 0
- Forks
- 1
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
Since in AI4E we plan on invoking LLMs via API endpoints for the time being, we need to implement this logic in the app.
The question is what API we will be using as every API may need to be processed differently. For now we should assume the API we will be invoking is also accessible from Langchain, thus making the necessary changes minimal.
As third-party API endpoints for LLM inovcations are not free, we need to make sure to limit the number of requests as well as their content. To this end we should ideally set up **authentication**, we should use AIoD keycloak. However this may not be feasible to be implemented within this release due to lack of, in which case we may need to resort to implementing more generic throttling instead.
Ideally, each user would have their own LLM invocation limits imposed on them, but since we are likely to skip integrating authentication for now, we need to limit the POST request for performing filtered semantic search instead, e.g,, by defining the the permitted total number of API requests per minute/hour that could be invoked.
More specifically, we need to implement the following precautions:
- [ ] Limit the number of `experimental/filtered_search` POST requests per some time period
- [ ] Each such request can invoke an LLM API endpoint for the total of N times
- [ ] User query in this request is limited by char count and sanitized properly
- [ ] Limit the number of characters generated by the LLM
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Bewertung
Dieses Issue wurde noch nicht bewertet.