MaartenGr / MaartenGr/PolyFuzz

Issue Facing While Fitting The Model With Huge Data

Open
#64 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
803
Forks
72
PR merge metrics
No merged PRs in 30d

Description

I have data contains around **166793 Records**, I want to fit this records for TF-IDF Model
```python
from polyfuzz.models import TFIDF
from polyfuzz import PolyFuzz

data=[] # Data contains total **166793 Records**
tfidf = TFIDF(n_gram_range=(1, 1), model_id="TF-IDF")
model = PolyFuzz(tfidf)

model.fit(data)
```

Here i am facing the issue while fitting the model ,The server getting killed (I have tried with configuration of **20 gb ram**).
Is there any solution?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the TFIDF model and the PolyFuzz model.fit(data) call shown in the issue, reproducing it with roughly 166793 records and the reported 20 GB configuration. Measure where memory is consumed during fitting and check whether the project documents a supported approach for datasets of this size; done means identifying a reproducible cause and a verified workaround or limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.