aws / aws/sagemaker-python-sdk

Unable to increase context length in SM serverless customizations

Open
#5,665 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

**Describe the feature you'd like**
Currently customers cannot use GRPO for Qwen model customization for models using long context (more than 2048 tokens)

**How would this feature be used? Please describe.**
Should work similar to JumpStart training implementation:
estimator = JumpStartEstimator(
model_id="meta-textgeneration-llama-2-7b",
hyperparameters={
"max_input_length": "4096", # Update context length here
"max_total_tokens": "4096"
}
)

**Describe alternatives you've considered**
Using SM training jobs for GRPO training with recipes.

**Additional context**
Customer: Intuit (a high-priority issue blocking usage of SM serverless customization service).

Contributor guide

Open the contributing guide

Research direction

Start with the serverless customization path and compare its configuration with the JumpStartEstimator example in the issue. Trace how GRPO customization handles Qwen models and the max_input_length and max_total_tokens hyperparameters. Done means customers can use those settings for context lengths above 2048 tokens in SM serverless customization.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.