AOSSIE-Org / AOSSIE-Org/Agora-Blockchain

[Enhancement]: Chatbot Improvement Refinements

Offen
#198 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
JavaScript
Sterne
97
Forks
199
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

### Is there an existing issue for this?

- [x] I have searched the existing issues

## Issue Summary

Referenced Folder: [Chatbot](https://github.com/AOSSIE-Org/Agora-Blockchain/blob/main/chatbot/)
While reviewing the codebase, I found some possible improvements that can make the chatbot model perform better. This proposal aims to enhance the chatbot's stability, generalization, and performance by standardizing the training and deployment processes.

1. NLP Standardization
The project uses very simple and naive tokenizer and stemmer. We can surely make it better by using the standard ones.
- Use nltk.word_tokenize for superior tokenization.
- Use PorterStemmer for effective stemming.

2. Training Refinements (Efficiency & Monitoring)
- Logging: Calculate and print the Average Epoch Loss (instead of just the final batch loss).
- Performance: Set num_workers > 0 in the DataLoader for parallel data fetching.

If I find any more possible improvements, I will update the PR accordingly.

### Record

- [x] I have synced all my node versions as mentioned in the project
- [x] I am using the same version of npm as is the project
- [x] My current branch is in sync with the development branch
- [x] I want to work on this issue

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.