ModelEngine-Group / ModelEngine-Group/nexent
[Bug] `calculate_term_weights` divides by zero when input is all stop words
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 731
- Avg merge
- 19h 34m
- Merged PRs (30d)
- 172
Description
Quick one. In sdk/nexent/core/nlp/tokenizer.py:59:
tf_weights = {term: weight / total_weight for term, weight in term_stats.items()}
total_weight is only incremented inside the if word not in analyse.default_tfidf.stop_words and word.strip() branch (lines 46-56). If every token is a stop word (e.g., a query consisting only of punctuation, whitespace, or common particles like "的 是 在"), total_weight == 0.0 and term_stats is empty.
In that case the dict comprehension is empty (no division), so today the function silently returns {} from the meaningful-terms guard at line 98 — but the moment any non-stop token slips in, the division will execute, and any future change that pre-populates term_stats without updating total_weight will raise ZeroDivisionError. The dependency between the two accumulators is fragile and undocumented.
Suggested fix: guard explicitly.
if total_weight == 0.0:
return {}
tf_weights = {term: weight / total_weight for term, weight in term_stats.items()}
This makes the contract obvious and prevents the latent footgun.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in sdk/nexent/core/nlp/tokenizer.py at calculate_term_weights around line 59, and inspect how total_weight and term_stats are built. Exercise the function with all-stop-word input and with a non-stop token; done means zero-weight input returns {} without ZeroDivisionError and normal weighting still works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Bug
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100