ModelEngine-Group / ModelEngine-Group/nexent

[Bug] `calculate_term_weights` divides by zero when input is all stop words

Open Beginner friendly
#3,813 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5.9k
Forks
731
Avg merge
19h 34m
Merged PRs (30d)
172

Description

Quick one. In sdk/nexent/core/nlp/tokenizer.py:59:

tf_weights = {term: weight / total_weight for term, weight in term_stats.items()}

total_weight is only incremented inside the if word not in analyse.default_tfidf.stop_words and word.strip() branch (lines 46-56). If every token is a stop word (e.g., a query consisting only of punctuation, whitespace, or common particles like "的 是 在"), total_weight == 0.0 and term_stats is empty.

In that case the dict comprehension is empty (no division), so today the function silently returns {} from the meaningful-terms guard at line 98 — but the moment any non-stop token slips in, the division will execute, and any future change that pre-populates term_stats without updating total_weight will raise ZeroDivisionError. The dependency between the two accumulators is fragile and undocumented.

Suggested fix: guard explicitly.

if total_weight == 0.0:
    return {}
tf_weights = {term: weight / total_weight for term, weight in term_stats.items()}

This makes the contract obvious and prevents the latent footgun.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in sdk/nexent/core/nlp/tokenizer.py at calculate_term_weights around line 59, and inspect how total_weight and term_stats are built. Exercise the function with all-stop-word input and with a non-stop token; done means zero-weight input returns {} without ZeroDivisionError and normal weighting still works.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai
Issue type
Bug
Difficulty
1/5
Estimated time
Under an hour
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.