AOSSIE-Org / AOSSIE-Org/OpenVerifiableLLM

[FEATURE]: ZKP Integration for Merkle Proofs

未关闭
#36 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
enhancement
主要语言
Python
星标
18
派生
31
平均合并
1 分钟
30 天内合并 PR
2

描述

### Feature and its Use Cases

The framework highlights the need for mechanisms such as:

* Zero-Knowledge Proofs of Training (ZKPoT)
* Zero-Knowledge Proofs of Inference (ZKPoI)

# Problem

our preprocessing pipeline for Wikipedia data dumps generated Merkle proofs for data chunks, but output them in a flat, unstructured JSON format. While mathematically correct, this format was fundamentally incompatible with modern Zero-Knowledge Proof (ZKP) circuit systems.
Before this update, the function:

```
export_merkle_proof()
```
generated a flat JSON structure.

Previous Output Format
```
{
"chunk_index": 1,
"chunk_size": 1048576,
"proof": [
["e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", true],
["...", false]
]
}
```

what json look like now
```
{
"wikipedia_dump": "simplewiki-20260201-pages-articles-multistream.xml.bz2",
"dump_date": "2026-02-01",
"raw_sha256": "...",
"processed_sha256": "...",
"raw_merkle_root": "...",
"processed_merkle_root": "...",
"chunk_size_bytes": 1048576,
"preprocessing_version": "v1",
"python_version": "3.11.9"
}
```
* No chunk-level verification: Cannot verify individual dataset chunks.
* Unstructured verification inputs: Verification parameters not clearly organized.
* Poor automation support: Requires manual preprocessing for verification pipelines

* ZKP circuits require a clear distinction between public inputs (e.g., Merkle root, chunk index) and witness data (e.g., sibling hashes). The previous flat JSON structure did not enforce this separation, forcing developers to manually parse and split the data.

* Modern proving systems like ```Halo2```, ```PLONK```, and ```Groth16``` expect structured inputs such as public_inputs and witness. The earlier format required additional preprocessing before it could be used in ZKP circuits.

# Fix
* ill redesign the ```export_merkle_proof() ```function in ```openverifiablellm/utils.py``` to output a structured JSON format that clearly separates public_inputs (e.g., chunk_index, chunk_size, merkle_root) and witness data (sibling_hashes). This format aligns with the input structure expected by ZKP frameworks such as``` Halo2, PLONK, and Groth16```.

* Update ```verify_merkle_proof_from_file()``` to correctly parse the new structure

# Future Work releted to this
* Native ZKP circuit integration(for Halo2, PLONK, and Groth16 )
* ZK Proof of Dataset Inclusion and End-to-End ZKPoT Pipeline

### Additional Context

_No response_

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。