aws / aws/fmeval

[Feature] Lambda model runner

Open
#331 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
291
Forks
60
PR merge metrics
No merged PRs in 30d

Description

While LLMs will not be hosted directly on lambdas, allowing customers to proxy their eval requests through Lambda invocations would expand the breadth of use cases of FMEval, without needing to maintain a vast set of model runner classes. Use cases include:

1. Use cases where customers host LLMs hosted on bare metal (EC2, etc).
2. Use cases where LLMs that are hosted on Sagemaker or Bedrock, but invoked through a web service that performs preprocessing / post-processing.

The Lambda model runner can function identically to the Bedrock model runner, replacing the [Bedrock runtime call](https://github.com/aws/fmeval/blob/main/src/fmeval/model_runners/bedrock_model_runner.py#L67-L69) with a lambda invocation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.