Provide batching alternative to FieldToEmbedding SMT for LLM integrations [DBZ-9182]
Open
component/core-library
migrated-from-jira
type/enhancement
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-9182](https://issues.redhat.com/browse/DBZ-9182)
The `FieldToEmbedding` SMT, by the nature of being a _single_ message transform, processes elements one-by-one, which can be detrimental to latency and throughput in particular when also supporting remote LLMs. Let's explore an alternative contract which allows to process records in batches, thus doing a single LLM invocation for calculating embeddings for an entire batch (provided of course a model supports this, but many do), and amortizing the cost of the call.
Contributor guide
Assessment
This issue has not been assessed yet.