Connection aborted. RemoteDisconnected('Remote end closed connection without response')
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 491
- Forks
- 290
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 3
Description
Work environment
| Questions | Answers |
|---|---|
| Type of issue | Bug |
| OS version (server) | Ubuntu I believe |
| OS version (client) | MacOS |
| PHP version | 7.2 |
| MISP version | 2.4.133 |
| PyMisp version | 2.4.133 |
In Dockerfile
RUN sed -i "s/max_execution_time = 30/max_execution_time = 300/" /etc/php/7.2/apache2/php.ini ; \
sed -i "s/memory_limit = 128M/memory_limit = 8192M/" /etc/php/7.2/apache2/php.ini ; \
sed -i "s/upload_max_filesize = 2M/upload_max_filesize = 50M/" /etc/php/7.2/apache2/php.ini ; \
sed -i "s/post_max_size = 8M/post_max_size = 50M/" /etc/php/7.2/apache2/php.ini
In MISP ui
max_execution_time (300)… OK (recommended: 300)
memory_limit (8192M)… OK (recommended: 2048M)
upload_max_filesize (50M)… OK (recommended: 50M)
post_max_size (50M)… OK (recommended: 50M)
Expected behavior
I have purpose written code which locates events with specific tags, creates new tags, tags all found events with new tags, republishes those events, & then deletes the old tags. Essentially I am updating tags on events & removing old tags.
Actual behavior
Some tags have over 9,000+ events associated with them. When I encounter such tags, the code goes about its normal execution. As the code attempts to search for all the events for a particular tag with previously statement total, I get this error message
{"asctime": "2021-01-19 21:46:07,358", "timestamp": 1611092767.394494, "name": "misp-feeds", "filename": "search_and_destroy.py", "funcName": "_search", "lineno": 228, "level": "ERROR", "levelno": 40, "message": "Searching for events failed. kwargs: {'controller': 'events', 'tags': 'malware:GandCrab', 'limit': 500, 'date_to': 1611092482, 'published': True, 'page': 4} Reason: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))", "severity": "ERROR"}
I tried "streaming" events by limiting how many events are returned by MISP in a while loop, cycling through until there are no events returned
old_events = client._search(controller='events', tags=old_tag_name,
limit=500, date_to=now, published=True)
However, I get the same error. I had a tag with 9,015 event. My code processed exactly 1,500 events and then refuses to continue even after I stop the script and run it again.
Steps to reproduce the behavior
I looked through other issues related to this and found a similar issue https://github.com/MISP/MISP/issues/6805. Im not exactly sure how to reproduce this.
Logs, screenshots, configuration dump, ...

Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the caller's search_and_destroy.py _search path and PyMISP client._search, focusing on paginated event searches with tags, limit 500, date_to, published, and page 4. Compare the behavior with issue #6805 and reproduce against a tag with thousands of events; done means the search completes without the RemoteDisconnected failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php, python
- Domain
- api
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100