AOSSIE-Org / AOSSIE-Org/Agora-Blockchain

[Enhancement]: Transition to Semantic Embeddings and NLTK Lemmatization

Offen
#238 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
JavaScript
Sterne
97
Forks
199
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

### Is there an existing issue for this?
No

- [x] I have searched the existing issues

### Issue Description ✍️

## 📌 Issue Summary
The current implementation of the Agora chatbot relies on a Bag of Words (BoW) model with basic string splitting for tokenization. This approach leads to several limitations:

1. The bot only understands words exactly as they appear in the training data. For instance, if the model is trained on the word "delivery" and a user types "delivering", the bot fails to recognize it as the same intent due to the absence of lemmatization.

2. There is no understanding of semantic relationships between words. Words such as "Hi", "Hello" and "Hey" are treated as completely different and mathematically unrelated tokens, just as unrelated as "Hi" and "Television" are.

3. As the vocabulary grows, the size of the input vector increases proportionally, resulting in high memory usage, slower performance and exposure to the curse of dimensionality, which negatively impacts scalability and efficiency.

---

## Solution

The proposed solution is to upgrade the existing NLP pipeline by integrating NLTK or SpaCy for advanced preprocessing and Sentence-Transformers (BERT-based models) for generating dense semantic embeddings.

1. This upgrade will introduce robustness to linguistic variations, allowing the chatbot to handle synonyms, different verb tenses and semantically similar phrases without breaking intent detection.

2. Instead of sparse and ever-growing BoW vectors, the model will use fixed-size dense embeddings, ensuring that input dimensionality remains constant regardless of vocabulary size.

3. By leveraging transfer learning from large-scale pre-trained transformer models, the chatbot will achieve higher accuracy and improved contextual understanding compared to the current custom-built BoW approach.

---

I am willing to work on this issue and integrate the suggested enhancement into the project.
Therefore, I kindly request the mentor to assign this issue to me under GSoC 2026.

Regards

### Record

- [x] I have synced all my node versions as mentioned in the project
- [x] I am using the same version of npm as is the project
- [x] My current branch is in sync with the development branch
- [x] I want to work on this issue

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.