AOSSIE-Org / AOSSIE-Org/OpenVerifiableLLM
[FEATURE]: ZKP Integration for Merkle Proofs
- Ngôn ngữ chính
- Python
- Star
- 18
- Fork
- 31
- Merge trung bình
- 1 phút
- Pull request đã merge (30 ngày)
- 2
Mô tả
### Feature and its Use Cases
The framework highlights the need for mechanisms such as:
* Zero-Knowledge Proofs of Training (ZKPoT)
* Zero-Knowledge Proofs of Inference (ZKPoI)
# Problem
our preprocessing pipeline for Wikipedia data dumps generated Merkle proofs for data chunks, but output them in a flat, unstructured JSON format. While mathematically correct, this format was fundamentally incompatible with modern Zero-Knowledge Proof (ZKP) circuit systems.
Before this update, the function:
```
export_merkle_proof()
```
generated a flat JSON structure.
Previous Output Format
```
{
"chunk_index": 1,
"chunk_size": 1048576,
"proof": [
["e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", true],
["...", false]
]
}
```
what json look like now
```
{
"wikipedia_dump": "simplewiki-20260201-pages-articles-multistream.xml.bz2",
"dump_date": "2026-02-01",
"raw_sha256": "...",
"processed_sha256": "...",
"raw_merkle_root": "...",
"processed_merkle_root": "...",
"chunk_size_bytes": 1048576,
"preprocessing_version": "v1",
"python_version": "3.11.9"
}
```
* No chunk-level verification: Cannot verify individual dataset chunks.
* Unstructured verification inputs: Verification parameters not clearly organized.
* Poor automation support: Requires manual preprocessing for verification pipelines
* ZKP circuits require a clear distinction between public inputs (e.g., Merkle root, chunk index) and witness data (e.g., sibling hashes). The previous flat JSON structure did not enforce this separation, forcing developers to manually parse and split the data.
* Modern proving systems like ```Halo2```, ```PLONK```, and ```Groth16``` expect structured inputs such as public_inputs and witness. The earlier format required additional preprocessing before it could be used in ZKP circuits.
# Fix
* ill redesign the ```export_merkle_proof() ```function in ```openverifiablellm/utils.py``` to output a structured JSON format that clearly separates public_inputs (e.g., chunk_index, chunk_size, merkle_root) and witness data (sibling_hashes). This format aligns with the input structure expected by ZKP frameworks such as``` Halo2, PLONK, and Groth16```.
* Update ```verify_merkle_proof_from_file()``` to correctly parse the new structure
# Future Work releted to this
* Native ZKP circuit integration(for Halo2, PLONK, and Groth16 )
* ZK Proof of Dataset Inclusion and End-to-End ZKPoT Pipeline
### Additional Context
_No response_
### Code of Conduct
- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.