aws / aws/amazon-sagemaker-feedback

SageMaker AI Fine-tuning: Misleading evaluation dataset format documentation

Open
#241 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
No language data
Stars
10
Forks
3
PR merge metrics
No merged PRs in 30d

Description

### Product Version

- [ ] Amazon SageMaker Studio Classic
- [x] Amazon SageMaker Studio
- [ ] Issue is not related to SageMaker Studio

### Issue Description

### Problem
Unable to upload evaluation datasets for fine-tuning in SageMaker AI. All formats fail validation with error: "Invalid dataset. Does not match any expected format."

### What We Tried

**Format 1: UI Examples (with system prompt)**
The UI shows this format:
```json
{
"system": "You are an English major with top marks in class who likes to give minimal word responses: ",
"query": "What is the symbol that ends the sentence as a question",
"response": "?"
}
```
**Result**: "Invalid dataset. Does not match any expected format"

**Format 2: UI Examples (single line JSONL)**
```json
{"system": "You are an English major...", "query": "What is the symbol...", "response": "?"}
```
**Result**: "Invalid dataset. Does not match any expected format"

**Format 3: Documentation Format (prompt/referenceResponse)**
Based on https://docs.aws.amazon.com/sagemaker-unified-studio/latest/userguide/model-evaluation-prompt-datasets-custom.html:
```json
{"prompt": "What is the symbol that ends the sentence as a question", "referenceResponse": "?"}
```
**Result**: "Invalid dataset. Does not match any expected format"

**Format 4: Pretty-printed JSON**
Multi-line formatted JSON objects with blank lines between
**Result**: "Invalid JSON format in file"

**Format 5: JSONL without blank lines**
Consecutive JSON objects, one per line
**Result**: "Invalid dataset. Does not match any expected format"

Image

### Steps to Reproduce
1. Go to SageMaker AI fine-tuning interface
2. Navigate to evaluation dataset upload
3. Create a .jsonl file with ANY of the formats above (including exact UI examples)
4. Attempt to upload
5. Receive error: "Invalid dataset. Does not match any expected format"

### Request
Please provide:
1. A working example file that passes validation
2. Complete format specification with all required/optional fields
3. Fix for the upload validator or update documentation to match actual requirements

### Expected Behavior

### Expected Behavior
- At minimum, the exact examples shown in the UI should work
- OR the documentation format should work
- OR there should be clear error messages explaining what's wrong

### Observed Behavior

### Actual Behavior
- **Every format fails validation**
- Even copying the exact examples from the UI fails
- No specific error message about what's wrong with the format
- Generic error provides no actionable guidance

### Impact
- **Blocker**: Cannot upload evaluation datasets at all
- Affects both custom evaluation datasets and attempts to use documented formats
- No workaround available
- Makes fine-tuning evaluation not possible

### Product Category

Models

### Feedback Category

Configuration and Setup

### Other Details

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the upload failures with the five formats listed in the issue, then compare the SageMaker Studio fine-tuning validator with the linked evaluation dataset documentation and the UI examples. Done means a documented working example and format specification, or aligned validation behavior with actionable error messages.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.