ModelEngine-Group / ModelEngine-Group/nexent

ContentClassifier: `MAX_BUFFER_SIZE` is declared but never enforced

Open Beginner friendly
#3,373 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5.9k
Forks
731
Avg merge
19h 34m
Merged PRs (30d)
172

Description

backend/utils/content_classifier_utils.py:20 advertises a 1 MB buffer cap as part of the "DoS protection" the class docstring touts:

MAX_BUFFER_SIZE = 1024 * 1024  # 1MB
MAX_TAG_LENGTH = 256           # Single tag max length
MAX_PATH_LENGTH = 512          # File path max length
MAX_TAG_COUNT = 100            # Max tags before stopping

A search through the same file shows the only places that mention these constants:

20:    MAX_BUFFER_SIZE = 1024 * 1024  # 1MB
23:    MAX_TAG_COUNT = 100            # Max tags before stopping
73:        if self.tag_count >= self.MAX_TAG_COUNT:

MAX_TAG_LENGTH and MAX_PATH_LENGTH are referenced (lines 63, 146); MAX_BUFFER_SIZE is never read. The classify() method (line 39) appends every incoming chunk to self.buffer with no upper bound:

def classify(self, chunk: str) -> List[Dict[str, Any]]:
    results = []
    self.buffer += chunk
    while self.buffer:
        ...

A streaming LLM that emits a very long run of "<" characters (or simply a very long single line with no recognised tag start) will grow self.buffer unboundedly while _process_non_tag_content emits 64-byte chunks back. An adversarial or buggy provider could pin memory.

Suggested fix

In classify():

self.buffer += chunk
if len(self.buffer) > self.MAX_BUFFER_SIZE:
    overflow = self.buffer[:self.MAX_BUFFER_SIZE]
    self.buffer = self.buffer[self.MAX_BUFFER_SIZE:]
    logger.warning("ContentClassifier buffer exceeded MAX_BUFFER_SIZE; truncating")
    # flush overflow as "others"

Either enforce the cap or delete the constant and the "DoS protection" claim in the docstring.

Category: B (error handling / resource exhaustion). Severity: Medium.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in backend/utils/content_classifier_utils.py, especially ContentClassifier.classify() and the MAX_BUFFER_SIZE declaration. Trace how self.buffer and _process_non_tag_content handle long unrecognized input, then enforce the documented cap or remove the unused constant and DoS-protection claim. Done means the buffer cannot grow beyond the intended limit and the chosen behavior is covered by the existing classifier flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, security
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.