AISocietyIITJ / AISocietyIITJ/FileNest

[FEATURE] Implement AI/ML Embedding & Multi-Depth Routing for .txt files

Open
#54 0 comments 0 reactions 2 assignees Claimed by @Anarghya1610 View on GitHub
enhancement
Dominant language
No language data
Stars
11
Forks
8
PR merge metrics
No merged PRs in 30d

Description

## 💡 What's your idea?

- Build the basic AI/ML pipeline for this project
- Start with .txt file embedding generation and multi-depth routing via clustering
- Integrate FAISS-based similarity search at Depth 1, and hierarchical routing till Depth 4
- Implement peertable-based routing and dynamic Tagging Vector (TV) creation

## 🤔 Why would this be useful?

- Forms the core semantic intelligence of FileNest
- Enables multi-depth decentralized storage and retrieval of file embeddings
- Provides foundation for multi-modal (image, video, pdf) support in future
- Decentralized, scalable, and AI-powered — as per the project proposal

## 🎯 Where should this feature go?

- [x] 🧠 AI/ML Models (AI components and machine learning)
- [ ] � Backend (Core server logic and APIs)
- [ ] 🌐 Frontend (CLI & React.js user interface)
- [ ] 🌐 Network Layer (P2P networking and communication)
- [ ] 🔗 Shared Utilities (Common code and libraries)
- [ ] 📚 Documentation
- [ ] 🛠️ Development tools

## 🖼️ How should it look/work?

- Peers generate text embeddings
- FAISS at Depth 1 finds nearest D1TV → routes embedding to selected peer
- At each next depth:
Peer performs similarity check (cosine similarity) against its TVs
If no match (similarity < threshold), assign new TV to a peer
Continue till Depth 4, where metadata is stored
- Each peer maintains a peertable.json:
Tracks its child TVs at one lower depth
Contains available peers list for assigning new TVs
- When creating a new TV:
Pick peer from available list/DHT (if available)
Send CreateTV message with centroid, TV ID, parent ID, thresholds, timestamp

## 🌟 How important is this feature?

- [x] 🔥 Critical - I really need this to use the app
- [ ] 🎯 High - This would make the app much better
- [ ] 😊 Medium - This would be nice to have
- [ ] 🤷 Low - Just a small improvement

## 📚 Examples

## 🔧 Implementation Ideas (Optional)

- Implement embedding generation module for .txt files
- Build FAISS clustering and search for Depth 1
- Define and integrate peertable JSON schema and CRUD logic
- Implement routing logic for Depth 1 to Depth 4
- Build CreateTV message structure and handler
- Simulate multi-depth routing locally using multiple peer processes
- Integrate libp2p mDNS peer discovery (optionally DHT later)

## 📱💻 Additional Context

- Core part of AI/ML deliverables per FileNest RAID proposal
- Focused on text files initially; will extend to other content types later
- Cleanly modular, decentralized, and scalable architecture

---

### 🚀 Want to build this feature yourself?
Great! Comment "I'd like to work on this" and we'll help you get started. Check out our [Contributing Guide](../CONTRIBUTING.md) if you're new to contributing!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.