aws-samples / aws-samples/sample-genai-on-eks-starter-kit

Accelerate LLM inference with post-training weight and activation using AWQ and GPTQ

Open
#69 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
94
Forks
59
Avg merge
1d 16h
Merged PRs (30d)
8

Description

Adapt the following but for vllm on EKS: [Accelerating LLM inference with post-training weight and activation using AWQ and GPTQ on Amazon SageMaker AI](https://aws.amazon.com/blogs/machine-learning/accelerating-llm-inference-with-post-training-weight-and-activation-using-awq-and-gptq-on-amazon-sagemaker-ai/).

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked Amazon SageMaker AI article and comparing its AWQ/GPTQ approach with vLLM on Amazon EKS. Determine which existing vLLM and EKS entry points in this repository should be adapted, and define completion as a documented, working equivalent for that environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
ai, cloud, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.