MISP / MISP/PyMISP

Connection aborted. RemoteDisconnected('Remote end closed connection without response')

Open
#687 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

T: support
Dominant language
Python
Stars
491
Forks
290
Avg merge
2d 8h
Merged PRs (30d)
3

Description

Work environment
Questions Answers
Type of issue Bug
OS version (server) Ubuntu I believe
OS version (client) MacOS
PHP version 7.2
MISP version 2.4.133
PyMisp version 2.4.133

In Dockerfile

RUN sed -i "s/max_execution_time = 30/max_execution_time = 300/" /etc/php/7.2/apache2/php.ini ; \
    sed -i "s/memory_limit = 128M/memory_limit = 8192M/" /etc/php/7.2/apache2/php.ini ; \
    sed -i "s/upload_max_filesize = 2M/upload_max_filesize = 50M/" /etc/php/7.2/apache2/php.ini ; \
    sed -i "s/post_max_size = 8M/post_max_size = 50M/" /etc/php/7.2/apache2/php.ini

In MISP ui

max_execution_time (300)… OK (recommended: 300)
memory_limit (8192M)… OK (recommended: 2048M)
upload_max_filesize (50M)… OK (recommended: 50M)
post_max_size (50M)… OK (recommended: 50M)
Expected behavior

I have purpose written code which locates events with specific tags, creates new tags, tags all found events with new tags, republishes those events, & then deletes the old tags. Essentially I am updating tags on events & removing old tags.

Actual behavior

Some tags have over 9,000+ events associated with them. When I encounter such tags, the code goes about its normal execution. As the code attempts to search for all the events for a particular tag with previously statement total, I get this error message
{"asctime": "2021-01-19 21:46:07,358", "timestamp": 1611092767.394494, "name": "misp-feeds", "filename": "search_and_destroy.py", "funcName": "_search", "lineno": 228, "level": "ERROR", "levelno": 40, "message": "Searching for events failed. kwargs: {'controller': 'events', 'tags': 'malware:GandCrab', 'limit': 500, 'date_to': 1611092482, 'published': True, 'page': 4} Reason: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))", "severity": "ERROR"}

I tried "streaming" events by limiting how many events are returned by MISP in a while loop, cycling through until there are no events returned
old_events = client._search(controller='events', tags=old_tag_name, limit=500, date_to=now, published=True)
However, I get the same error. I had a tag with 9,015 event. My code processed exactly 1,500 events and then refuses to continue even after I stop the script and run it again.

Steps to reproduce the behavior

I looked through other issues related to this and found a similar issue https://github.com/MISP/MISP/issues/6805. Im not exactly sure how to reproduce this.

Logs, screenshots, configuration dump, ...

Screen Shot 2021-01-19 at 3 55 01 PM

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the caller's search_and_destroy.py _search path and PyMISP client._search, focusing on paginated event searches with tags, limit 500, date_to, published, and page 4. Compare the behavior with issue #6805 and reproduce against a tag with thousands of events; done means the search completes without the RemoteDisconnected failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
php, python
Domain
api
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.