aws / aws/amazon-sagemaker-feedback
SageMaker AI Fine-tuning: Misleading evaluation dataset format documentation
- Dominant language
- No language data
- Stars
- 10
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
### Product Version
- [ ] Amazon SageMaker Studio Classic
- [x] Amazon SageMaker Studio
- [ ] Issue is not related to SageMaker Studio
### Issue Description
### Problem
Unable to upload evaluation datasets for fine-tuning in SageMaker AI. All formats fail validation with error: "Invalid dataset. Does not match any expected format."
### What We Tried
**Format 1: UI Examples (with system prompt)**
The UI shows this format:
```json
{
"system": "You are an English major with top marks in class who likes to give minimal word responses: ",
"query": "What is the symbol that ends the sentence as a question",
"response": "?"
}
```
**Result**: "Invalid dataset. Does not match any expected format"
**Format 2: UI Examples (single line JSONL)**
```json
{"system": "You are an English major...", "query": "What is the symbol...", "response": "?"}
```
**Result**: "Invalid dataset. Does not match any expected format"
**Format 3: Documentation Format (prompt/referenceResponse)**
Based on https://docs.aws.amazon.com/sagemaker-unified-studio/latest/userguide/model-evaluation-prompt-datasets-custom.html:
```json
{"prompt": "What is the symbol that ends the sentence as a question", "referenceResponse": "?"}
```
**Result**: "Invalid dataset. Does not match any expected format"
**Format 4: Pretty-printed JSON**
Multi-line formatted JSON objects with blank lines between
**Result**: "Invalid JSON format in file"
**Format 5: JSONL without blank lines**
Consecutive JSON objects, one per line
**Result**: "Invalid dataset. Does not match any expected format"
### Steps to Reproduce
1. Go to SageMaker AI fine-tuning interface
2. Navigate to evaluation dataset upload
3. Create a .jsonl file with ANY of the formats above (including exact UI examples)
4. Attempt to upload
5. Receive error: "Invalid dataset. Does not match any expected format"
### Request
Please provide:
1. A working example file that passes validation
2. Complete format specification with all required/optional fields
3. Fix for the upload validator or update documentation to match actual requirements
### Expected Behavior
### Expected Behavior
- At minimum, the exact examples shown in the UI should work
- OR the documentation format should work
- OR there should be clear error messages explaining what's wrong
### Observed Behavior
### Actual Behavior
- **Every format fails validation**
- Even copying the exact examples from the UI fails
- No specific error message about what's wrong with the format
- Generic error provides no actionable guidance
### Impact
- **Blocker**: Cannot upload evaluation datasets at all
- Affects both custom evaluation datasets and attempts to use documented formats
- No workaround available
- Makes fine-tuning evaluation not possible
### Product Category
Models
### Feedback Category
Configuration and Setup
### Other Details
_No response_
Contributor guide
Research direction
Start by reproducing the upload failures with the five formats listed in the issue, then compare the SageMaker Studio fine-tuning validator with the linked evaluation dataset documentation and the UI examples. Done means a documented working example and format specification, or aligned validation behavior with actionable error messages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100