AOSSIE-Org / AOSSIE-Org/OpenVerifiableLLM

[FEATURE]: Add Diagnostic Verification Reporting (HTML, Markdown, and detailed failure diagnostics)

Open
#87 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
18
Forks
31
Avg merge
1m
Merged PRs (30d)
2

Description

### Feature and its Use Cases

Currently, when a user runs verify_dataset.py, the verification report is limited to:

An ASCII table printed to the terminal showing PASS/FAIL per check
An optional raw JSON file via the --json flag
This works for quick checks, but it falls short in two critical areas:

Problem 1: No shareable, readable reports
The ASCII table only makes sense in a terminal. If a researcher wants to share verification results with a team, attach them to a paper, or post them in a GitHub issue/PR — there's no way to do it. The JSON output is machine-readable but not human-friendly.

Problem 2: No diagnostic details when verification fails
When a check fails (e.g., processed_sha256 shows FAIL), the report only shows the expected vs actual hash. It gives zero guidance on why it failed. The user is left guessing:

Did the environment change? Which packages differ?
Did the processed output change? Where exactly does it diverge?
Is it a Python version issue? A platform issue?

### Additional Context

_No response_

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.