AOSSIE-Org / AOSSIE-Org/OpenVerifiableLLM

[FEATURE]: ZKP Integration for Merkle Proofs

Abierto
#36 1 comentario 0 reacciones 0 asignados Ver en GitHub
enhancement
Lenguaje dominante
Python
Estrellas
18
Forks
31
Merge medio
1 min
PR fusionados (30 d)
2

Descripción

### Feature and its Use Cases

The framework highlights the need for mechanisms such as:

* Zero-Knowledge Proofs of Training (ZKPoT)
* Zero-Knowledge Proofs of Inference (ZKPoI)

# Problem

our preprocessing pipeline for Wikipedia data dumps generated Merkle proofs for data chunks, but output them in a flat, unstructured JSON format. While mathematically correct, this format was fundamentally incompatible with modern Zero-Knowledge Proof (ZKP) circuit systems.
Before this update, the function:

```
export_merkle_proof()
```
generated a flat JSON structure.

Previous Output Format
```
{
"chunk_index": 1,
"chunk_size": 1048576,
"proof": [
["e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", true],
["...", false]
]
}
```

what json look like now
```
{
"wikipedia_dump": "simplewiki-20260201-pages-articles-multistream.xml.bz2",
"dump_date": "2026-02-01",
"raw_sha256": "...",
"processed_sha256": "...",
"raw_merkle_root": "...",
"processed_merkle_root": "...",
"chunk_size_bytes": 1048576,
"preprocessing_version": "v1",
"python_version": "3.11.9"
}
```
* No chunk-level verification: Cannot verify individual dataset chunks.
* Unstructured verification inputs: Verification parameters not clearly organized.
* Poor automation support: Requires manual preprocessing for verification pipelines

* ZKP circuits require a clear distinction between public inputs (e.g., Merkle root, chunk index) and witness data (e.g., sibling hashes). The previous flat JSON structure did not enforce this separation, forcing developers to manually parse and split the data.

* Modern proving systems like ```Halo2```, ```PLONK```, and ```Groth16``` expect structured inputs such as public_inputs and witness. The earlier format required additional preprocessing before it could be used in ZKP circuits.

# Fix
* ill redesign the ```export_merkle_proof() ```function in ```openverifiablellm/utils.py``` to output a structured JSON format that clearly separates public_inputs (e.g., chunk_index, chunk_size, merkle_root) and witness data (sibling_hashes). This format aligns with the input structure expected by ZKP frameworks such as``` Halo2, PLONK, and Groth16```.

* Update ```verify_merkle_proof_from_file()``` to correctly parse the new structure

# Future Work releted to this
* Native ZKP circuit integration(for Halo2, PLONK, and Groth16 )
* ZK Proof of Dataset Inclusion and End-to-End ZKPoT Pipeline

### Additional Context

_No response_

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.