MISP / MISP/PyMISP

misp.search() filters aren't applied to the initial db query

Open
#751 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

S: needs more info T: support
Dominant language
Python
Stars
491
Forks
290
Avg merge
2d 8h
Merged PRs (30d)
3

Description

I'm using PyMISP's misp.search() to return records from a MISP instance containing 25mi+ events. Filters for misp.search() are imported from a settings.py file like this:

filters = {'published':'true','date':'2021-06-01','tags':['my_tag']}

and it is called like this:

events = misp.search(metadata=true, limit=entries, **filters, pythonify=True)

I get the records I'm looking for, but as the database grows I started to see more and more 'out of memory' errors. Investigating, I've found this query is executed in the database, regardless of the filters applied to misp.search():

SELECT Event.id, Event.org_id, Event.date, Event.info, Event.user_id, Event.uuid, Event.published, Event.analysis, Event.attribute_count, Event.orgc_id, Event.timestamp, Event.distribution, Event.sharing_group_id, Event.proposal_email_lock, Event.locked, Event.threat_level_id, Event.publish_timestamp, Event.sighting_timestamp, Event.disable_correlation, Event.extends_uuid FROM misp.events AS Event WHERE Event.user_id = (2)

That is not scalable as it basically loads the whole 'events' table to memory - as we use user_id = 2 to import everything into MISP.

What I don't understand is why MISP is running that query without any of the filter I've applied via misp_search(). It looks like it applies the filters after retrieving results and putting them in memory.

MISP 2.4.133
Python 7.2.24
PyMISP latest, just updated it

Perhaps I'm not applying the filters correctly? Anybody can reproduce this situation?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the misp.search() call with the published, date, and tags filters and inspect the request it sends to the MISP REST API. Compare the resulting request and server-side query behavior with the reported broad events query; done means the filters are applied before the result set is loaded into memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.