TheHive-Project / TheHive-Project/Cortex-Analyzers
[Bug] EmlParser's artifacts() method produces 'garbage' URLs when str(dict) is passed to iocextract.extract_urls()
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 490
- Forks
- 405
- Avg merge
- 2d 43m
- Merged PRs (30d)
- 8
Description
Describe the bug
At the moment we're passing the str(raw) to extractor methods of the iocextract. This causes some characters (like new-line, carriage-return) to be escaped by an additional backslash, which causes resulting artifacts to contain 'garbage'.
I'm attaching a patch, and also explaining on a simple code what's the issue and how to fix.
Let me know if this solution would be accepted, and I'll create a PR for that, and link it to this issue.
If you think this should be addressed in some other way -- please let me know as well :)
To Reproduce
Steps to reproduce the behavior:
- Have an observable *.eml with '\r\n' in the email body right after the URL that should be extracted (example email)
- Run the EmlParser on that email
- Observe that extracted URLs look like:
http://example.com/something/something\r\n\r\n\r\nWord
Expected behavior
In the example from above, the extracted URL should end before the \r\n sequence, and should look like this: http://example.com/something/something
Complementary information
This can be easily explained in the Python shell, on super-simple example:
>>> import iocextract
>>> # create a simple dictionary, where the value of a key contains '\r\n' sequences, like the EML files we want to parse in Cortex
...
>>> foo = {'k1': 'Here is https://some.url.com/blah\r\n\r\n\r\nAnd another one ftp://user:password@server.com/dir/dir2/dir3\r\n\r\nAnd IP:1.2.3.4\r\nAnd another: 8.8.8.8'}
>>> # now parse it the way we do it in the EmlParser right now
...
>>> list(iocextract.extract_urls(str(foo)))
['https://some.url.com/blah\\r\\n\\r\\n\\r\\nAnd', 'ftp://user:password@server.com/dir/dir2/dir3\\r\\n\\r\\nAnd']
>>> # as you see above, the parsed URLs contain garbage
...
>>> # now let's try to parse just the value
...
>>> list(iocextract.extract_urls(str(foo['k1'])))
['https://some.url.com/blah', 'ftp://user:password@server.com/dir/dir2/dir3']
>>> # as you see above, now the results are correct
...
Work environment
- Client OS: RHEL
- Server OS: RHEL
- Browse type and version: n/a
- Cortex version: 3.1
- Cortex Analyzer/Responder name: EmlParser
- Cortex Analyzer/Responder version: EmlParser version 1.2; repo git tag
2.9.4
Possible solutions
Instead of passing the whole dictionary as a string, we can iterate over the values of the dictionary. Just like shown above.
The patch that does this job, and is verified to be working:
iocextract_url_parsing_fix.txt
Additional context
Add any other context about the problem here.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the EmlParser artifacts() method and compare its current iocextract.extract_urls() input with the provided Python examples. Verify that parsing dictionary values avoids escaped carriage-return and newline text, then reproduce the sample URL extraction to confirm the resulting URLs end before those sequences.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- security
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100