AISocietyIITJ / AISocietyIITJ/FileNest
[FEATURE] Implement AI/ML Embedding & Multi-Depth Routing for .txt files
- Dominant language
- No language data
- Stars
- 11
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Description
## 💡 What's your idea?
- Build the basic AI/ML pipeline for this project
- Start with .txt file embedding generation and multi-depth routing via clustering
- Integrate FAISS-based similarity search at Depth 1, and hierarchical routing till Depth 4
- Implement peertable-based routing and dynamic Tagging Vector (TV) creation
## 🤔 Why would this be useful?
- Forms the core semantic intelligence of FileNest
- Enables multi-depth decentralized storage and retrieval of file embeddings
- Provides foundation for multi-modal (image, video, pdf) support in future
- Decentralized, scalable, and AI-powered — as per the project proposal
## 🎯 Where should this feature go?
- [x] 🧠 AI/ML Models (AI components and machine learning)
- [ ] � Backend (Core server logic and APIs)
- [ ] 🌐 Frontend (CLI & React.js user interface)
- [ ] 🌐 Network Layer (P2P networking and communication)
- [ ] 🔗 Shared Utilities (Common code and libraries)
- [ ] 📚 Documentation
- [ ] 🛠️ Development tools
## 🖼️ How should it look/work?
- Peers generate text embeddings
- FAISS at Depth 1 finds nearest D1TV → routes embedding to selected peer
- At each next depth:
Peer performs similarity check (cosine similarity) against its TVs
If no match (similarity < threshold), assign new TV to a peer
Continue till Depth 4, where metadata is stored
- Each peer maintains a peertable.json:
Tracks its child TVs at one lower depth
Contains available peers list for assigning new TVs
- When creating a new TV:
Pick peer from available list/DHT (if available)
Send CreateTV message with centroid, TV ID, parent ID, thresholds, timestamp
## 🌟 How important is this feature?
- [x] 🔥 Critical - I really need this to use the app
- [ ] 🎯 High - This would make the app much better
- [ ] 😊 Medium - This would be nice to have
- [ ] 🤷 Low - Just a small improvement
## 📚 Examples
## 🔧 Implementation Ideas (Optional)
- Implement embedding generation module for .txt files
- Build FAISS clustering and search for Depth 1
- Define and integrate peertable JSON schema and CRUD logic
- Implement routing logic for Depth 1 to Depth 4
- Build CreateTV message structure and handler
- Simulate multi-depth routing locally using multiple peer processes
- Integrate libp2p mDNS peer discovery (optionally DHT later)
## 📱💻 Additional Context
- Core part of AI/ML deliverables per FileNest RAID proposal
- Focused on text files initially; will extend to other content types later
- Cleanly modular, decentralized, and scalable architecture
---
### 🚀 Want to build this feature yourself?
Great! Comment "I'd like to work on this" and we'll help you get started. Check out our [Contributing Guide](../CONTRIBUTING.md) if you're new to contributing!
Contributor guide
Assessment
This issue has not been assessed yet.