jxnl / jxnl/instructor-classify

Add MarkdownReporter for visualization embedding

Open
#14 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
37
Forks
4
PR merge metrics
No merged PRs in 30d

Description

## Feature Request: MarkdownReporter

### Description
Add a dedicated MarkdownReporter to the modular evaluation framework that generates a comprehensive markdown file with embedded visualizations.

### Goals
- Create a new reporter in the reporting module that specializes in markdown generation
- Automatically embed all generated visualizations (confusion matrices, bootstrap distributions, etc.)
- Format results in markdown tables for better readability
- Generate a single, self-contained markdown file that can be viewed directly in GitHub or any markdown viewer
- Include all metrics, comparisons, and analysis results in a well-structured document

### Implementation Details
- Should extend the base Reporter class in eval_harness/base.py
- Should integrate with the existing visualization generation in the analyzers
- Need to modify the image paths to be relative in the markdown document
- Add to the configurable reporters in EvaluationConfig

### Benefits
- Better sharing of results with team members
- Easier to include evaluation results in documentation
- More accessible visualization of complex analysis
- Improved workflow for reviewing model performance

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.