ModelEngine-Group / ModelEngine-Group/nexent
ContentClassifier: `MAX_BUFFER_SIZE` is declared but never enforced
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 731
- Avg merge
- 19h 34m
- Merged PRs (30d)
- 172
Description
backend/utils/content_classifier_utils.py:20 advertises a 1 MB buffer cap as part of the "DoS protection" the class docstring touts:
MAX_BUFFER_SIZE = 1024 * 1024 # 1MB
MAX_TAG_LENGTH = 256 # Single tag max length
MAX_PATH_LENGTH = 512 # File path max length
MAX_TAG_COUNT = 100 # Max tags before stopping
A search through the same file shows the only places that mention these constants:
20: MAX_BUFFER_SIZE = 1024 * 1024 # 1MB
23: MAX_TAG_COUNT = 100 # Max tags before stopping
73: if self.tag_count >= self.MAX_TAG_COUNT:
MAX_TAG_LENGTH and MAX_PATH_LENGTH are referenced (lines 63, 146); MAX_BUFFER_SIZE is never read. The classify() method (line 39) appends every incoming chunk to self.buffer with no upper bound:
def classify(self, chunk: str) -> List[Dict[str, Any]]:
results = []
self.buffer += chunk
while self.buffer:
...
A streaming LLM that emits a very long run of "<" characters (or simply a very long single line with no recognised tag start) will grow self.buffer unboundedly while _process_non_tag_content emits 64-byte chunks back. An adversarial or buggy provider could pin memory.
Suggested fix
In classify():
self.buffer += chunk
if len(self.buffer) > self.MAX_BUFFER_SIZE:
overflow = self.buffer[:self.MAX_BUFFER_SIZE]
self.buffer = self.buffer[self.MAX_BUFFER_SIZE:]
logger.warning("ContentClassifier buffer exceeded MAX_BUFFER_SIZE; truncating")
# flush overflow as "others"
Either enforce the cap or delete the constant and the "DoS protection" claim in the docstring.
Category: B (error handling / resource exhaustion). Severity: Medium.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in backend/utils/content_classifier_utils.py, especially ContentClassifier.classify() and the MAX_BUFFER_SIZE declaration. Trace how self.buffer and _process_non_tag_content handle long unrecognized input, then enforce the documented cap or remove the unused constant and DoS-protection claim. Done means the buffer cannot grow beyond the intended limit and the chosen behavior is covered by the existing classifier flow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, security
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100