Diagnostic Dataset for Probing Visual Hallucinations in VLMs

### Summary

I've created a diagnostic dataset designed to systematically probe for hallucination behaviors in Molmo and other VLMs. This addresses the gap in standardized evaluation for visual hallucinations - a critical issue for VLM reliability.

### Motivation

Current VQA benchmarks (VQA v2, A-OKVQA, ChartQA, etc.) measure overall accuracy but don't specifically target hallucination patterns. This leaves systematic failure modes hidden in aggregate metrics.

This diagnostic dataset tests for:
- **Object existence/absence**: Does the model hallucinate objects that aren't there?
- **Attribute fabrication**: Does the model invent colors, brands, or properties?
- **Text hallucination**: Does the model fabricate unreadable text?
- **Counting errors**: Does the model give confident wrong counts?
- **Spatial errors**: Does the model hallucinate spatial relationships?
- **Language prior dominance**: Does the model answer from statistics vs vision?

### Dataset Structure

```
Total Test Cases: 18 (initial version)

By Hallucination Type:
object_existence: 2
object_absence: 1
attribute_fabrication: 3
count_error: 4
spatial_error: 2
text_fabrication: 2
language_prior: 2
identity_error: 2

By Difficulty: easy (3), medium (8), hard (7)
By Answer Type: binary (1), short (6), list (1), explanation (1), refusal (9)
Synthetic-only: 17 (can be generated without external images)
```

### Example Test Cases

#### Case 1: Object Absence (Presuppositional)
```python
{
"id": "obj_absence_003",
"question": "Describe the person in this image.",
"ground_truth": ["there is no person", "I don't see a person"],
"hallucination_type": "object_existence",
"expected_behavior": "Model should indicate no person is present",
"hallucination_pattern": "Model describes an imaginary person's appearance",
"rationale": "Presuppositional questions ('the person') prime models to hallucinate"
}
```

#### Case 2: Language Prior vs Visual Evidence
```python
{
"id": "lang_prior_001",
"question": "What color is the banana?",
"ground_truth": ["blue", "the banana is blue"],
"hallucination_type": "language_prior",
"expected_behavior": "Model should report the actual blue color shown",
"hallucination_pattern": "Model says 'yellow' based on prior knowledge"
}
```

#### Case 3: Attribute Fabrication (Grayscale)
```python
{
"id": "attr_fab_001",
"question": "What color is the car?",
"ground_truth": ["cannot determine", "the image is black and white"],
"hallucination_type": "attribute_fabrication",
"expected_behavior": "Should indicate color cannot be determined",
"hallucination_pattern": "Model confidently states a specific color"
}
```

### Design Principles

1. **Targeted**: Each case tests a specific hallucination type
2. **Annotated**: Includes expected behavior AND typical failure patterns
3. **Difficulty-graded**: Easy/Medium/Hard classifications
4. **Synthetic-first**: Most cases can use generated images (no external dependencies)
5. **Extensible**: Framework allows easy addition of new test cases

### Implementation

I've implemented this as a Python module following Molmo's dataset conventions:

```python
# Located at: olmo/data/hallucination_diagnostic_dataset.py

from olmo.data.hallucination_diagnostic_dataset import (
HALLUCINATION_DIAGNOSTIC_DATASET,
get_cases_by_type,
get_dataset_statistics,
HallucinationType,
)

# Get all object absence tests
absence_tests = get_cases_by_type(HallucinationType.OBJECT_ABSENCE)

# Export to JSON for external tools
export_to_json("hallucination_eval.json")
```

### Relationship to Existing Benchmarks

| Benchmark | Focus | This Dataset Adds |
|-----------|-------|-------------------|
| POPE | Object existence (binary) | Open-ended, presuppositional |
| CHAIR | Caption-level hallucination | QA-level, targeted probes |
| GAVIE | GPT-4 evaluation | Automatic, structured evaluation |
| PixMo-Count | Counting accuracy | Counting with zero/absence |

### Proposed Integration

1. **Standalone evaluation**: Can be used independently of training
2. **CI testing**: Quick sanity check for hallucination regression
3. **Model comparison**: Standardized protocol across model versions
4. **Research**: Publishable evaluation methodology

### Files Created

- `olmo/data/hallucination_diagnostic_dataset.py` - Main dataset definition
- `scripts/hallucination_test.py` - Runner for evaluation
- `hallucination_dataset_v0.1.json` - Exported JSON format

### Next Steps (Happy to Help With)

- [ ] Expand to 50+ test cases across all categories
- [ ] Generate synthetic test images
- [ ] Add evaluation metrics (hallucination rate, confidence calibration)
- [ ] Integrate with existing eval pipeline
- [ ] Write documentation

### Questions for Maintainers

1. Would this be useful to include in the evaluation suite?
2. Are there specific hallucination types you'd like prioritized?
3. Should this integrate with `launch_scripts/eval_downstream.py`?

### Related Work

- POPE: Polling-based Object Probing Evaluation (Li et al., 2023)
- CHAIR: Object Hallucination in Image Captioning (Rohrbach et al., 2018)
- Molmo paper section on evaluation limitations

@chrisc36 @sangho-vision @jamespark3922