MaartenGr / MaartenGr/PolyFuzz
Issue Facing While Fitting The Model With Huge Data
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 803
- Forks
- 72
- PR merge metrics
- No merged PRs in 30d
Description
I have data contains around **166793 Records**, I want to fit this records for TF-IDF Model
```python
from polyfuzz.models import TFIDF
from polyfuzz import PolyFuzz
data=[] # Data contains total **166793 Records**
tfidf = TFIDF(n_gram_range=(1, 1), model_id="TF-IDF")
model = PolyFuzz(tfidf)
model.fit(data)
```
Here i am facing the issue while fitting the model ,The server getting killed (I have tried with configuration of **20 gb ram**).
Is there any solution?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the TFIDF model and the PolyFuzz model.fit(data) call shown in the issue, reproducing it with roughly 166793 records and the reported 20 GB configuration. Measure where memory is consumed during fitting and check whether the project documents a supported approach for datasets of this size; done means identifying a reproducible cause and a verified workaround or limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100