misp.search() filters aren't applied to the initial db query
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 491
- Forks
- 290
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 3
Description
I'm using PyMISP's misp.search() to return records from a MISP instance containing 25mi+ events. Filters for misp.search() are imported from a settings.py file like this:
filters = {'published':'true','date':'2021-06-01','tags':['my_tag']}
and it is called like this:
events = misp.search(metadata=true, limit=entries, **filters, pythonify=True)
I get the records I'm looking for, but as the database grows I started to see more and more 'out of memory' errors. Investigating, I've found this query is executed in the database, regardless of the filters applied to misp.search():
SELECT Event.id, Event.org_id, Event.date, Event.info, Event.user_id, Event.uuid, Event.published, Event.analysis, Event.attribute_count, Event.orgc_id, Event.timestamp, Event.distribution, Event.sharing_group_id, Event.proposal_email_lock, Event.locked, Event.threat_level_id, Event.publish_timestamp, Event.sighting_timestamp, Event.disable_correlation, Event.extends_uuid FROM misp.events AS Event WHERE Event.user_id = (2)
That is not scalable as it basically loads the whole 'events' table to memory - as we use user_id = 2 to import everything into MISP.
What I don't understand is why MISP is running that query without any of the filter I've applied via misp_search(). It looks like it applies the filters after retrieving results and putting them in memory.
MISP 2.4.133
Python 7.2.24
PyMISP latest, just updated it
Perhaps I'm not applying the filters correctly? Anybody can reproduce this situation?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the misp.search() call with the published, date, and tags filters and inspect the request it sends to the MISP REST API. Compare the resulting request and server-side query behavior with the reported broad events query; done means the filters are applied before the result set is loaded into memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100