aws-samples / aws-samples/sample-genai-on-eks-starter-kit
Accelerate LLM inference with post-training weight and activation using AWQ and GPTQ
- Dominant language
- JavaScript
- Stars
- 94
- Forks
- 59
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 8
Description
Adapt the following but for vllm on EKS: [Accelerating LLM inference with post-training weight and activation using AWQ and GPTQ on Amazon SageMaker AI](https://aws.amazon.com/blogs/machine-learning/accelerating-llm-inference-with-post-training-weight-and-activation-using-awq-and-gptq-on-amazon-sagemaker-ai/).
Contributor guide
Research direction
Start by reading the linked Amazon SageMaker AI article and comparing its AWQ/GPTQ approach with vLLM on Amazon EKS. Determine which existing vLLM and EKS entry points in this repository should be adapted, and define completion as a documented, working equivalent for that environment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- ai, cloud, devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100