AOSSIE-Org / AOSSIE-Org/OpenVerifiableLLM
[FEATURE/ARCHITECTURE]: Implement Streaming Data Pipeline and Incremental Merkle Trees for OOM-Free, Scalable Dataset Verification
- Dominant language
- Python
- Stars
- 18
- Forks
- 31
- Avg merge
- 1m
- Merged PRs (30d)
- 2
Description
### **Description**
Currently, `OpenVerifiableLLM` extracts text from `.xml`/`.bz2` files and generates a Merkle Tree to ensure data verifiable integrity. However, as we scale to real-world LLM open-data sources (like full Wikipedia dumps or Common Crawl chunks that exceed 20GB+), loading these datasets entirely into memory to parse the XML and generate hashes will result in severe **Out-Of-Memory (OOM) crashes**.
Additionally, standard Merkle Tree implementations require a full recalculation of the tree whenever new documents are added to the corpus, creating a massive computational bottleneck for continuous pre-training or Retrieval-Augmented Generation (RAG) updates.
### **Proposed Solution**
I propose re-architecting the data ingestion and verification pipeline to be fully scalable, maintaining an $O(1)$ memory footprint and reducing update times to $O(\log N)$. We can achieve this through two major implementations:
**1. Event-Driven Streaming Data Parser**
* Refactor the `extract_text_from_xml` utility to use a streaming/event-driven approach (using Python's `xml.etree.ElementTree.iterparse` combined with generators).
* Instead of loading the entire DOM into memory, this will yield text chunks/documents one by one, allowing us to process multi-terabyte files smoothly without RAM spikes.
**2. Incremental (Append-Only) Merkle Tree Support**
* Extend the existing Merkle proof system to support dynamic appends.
* By maintaining the "Merkle Frontier" (caching only the rightmost unbalanced nodes at each depth), we can append new documents and calculate the new Verifiable Root Hash in milliseconds without having to re-hash the entire historical dataset.
### **Implementation Plan**
- [ ] **Step 1:** Create a `stream_text_from_xml()` generator in `utils.py` that processes `.bz2`/`.xml` files in chunks.
- [ ] **Step 2:** Update the hashing utility to compute leaf hashes on-the-fly as chunks are yielded by the generator.
- [ ] **Step 3:** Implement an `append_leaf()` method in the Merkle Tree class that mathematically handles incremental right-side appends.
- [ ] **Step 4:** Write unit tests comparing the Root Hash of a bulk-loaded tree vs. an incrementally-loaded tree (ensuring they match exactly).
- [ ] **Step 5:** Profile memory usage to mathematically prove the $O(1)$ memory consumption.
### **Impact and Value to the Project**
* **Production Readiness:** Transforms the extraction pipeline to support massive real-world open-weight datasets securely.
* **Efficiency:** Drastically reduces CPU overhead and totally eliminates memory bottlenecks.
* **True "Verifiable" Vision:** Enables continuous data updates (e.g., streaming live data into an LLM) while instantly maintaining verifiable proofs.
I would love to take ownership of this issue and start drafting a PR if the maintainers/mentors agree with this architectural direction! Let me know your thoughts.
Contributor guide
Assessment
This issue has not been assessed yet.