TheHive-Project / TheHive-Project/Cortex-Analyzers

[Bug] EmlParser's artifacts() method produces 'garbage' URLs when str(dict) is passed to iocextract.extract_urls()

Open
#944 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
490
Forks
405
Avg merge
2d 43m
Merged PRs (30d)
8

Description

Describe the bug
At the moment we're passing the str(raw) to extractor methods of the iocextract. This causes some characters (like new-line, carriage-return) to be escaped by an additional backslash, which causes resulting artifacts to contain 'garbage'.

I'm attaching a patch, and also explaining on a simple code what's the issue and how to fix.
Let me know if this solution would be accepted, and I'll create a PR for that, and link it to this issue.
If you think this should be addressed in some other way -- please let me know as well :)

To Reproduce
Steps to reproduce the behavior:

  1. Have an observable *.eml with '\r\n' in the email body right after the URL that should be extracted (example email)
  2. Run the EmlParser on that email
  3. Observe that extracted URLs look like: http://example.com/something/something\r\n\r\n\r\nWord

Expected behavior
In the example from above, the extracted URL should end before the \r\n sequence, and should look like this: http://example.com/something/something

Complementary information
This can be easily explained in the Python shell, on super-simple example:

>>> import iocextract
>>> # create a simple dictionary, where the value of a key contains '\r\n' sequences, like the EML files we want to parse in Cortex
...
>>> foo = {'k1': 'Here is https://some.url.com/blah\r\n\r\n\r\nAnd another one ftp://user:password@server.com/dir/dir2/dir3\r\n\r\nAnd IP:1.2.3.4\r\nAnd another: 8.8.8.8'}
>>> # now parse it the way we do it in the EmlParser right now
...
>>> list(iocextract.extract_urls(str(foo)))
['https://some.url.com/blah\\r\\n\\r\\n\\r\\nAnd', 'ftp://user:password@server.com/dir/dir2/dir3\\r\\n\\r\\nAnd']
>>> # as you see above, the parsed URLs contain garbage
...
>>> # now let's try to parse just the value
...
>>> list(iocextract.extract_urls(str(foo['k1'])))
['https://some.url.com/blah', 'ftp://user:password@server.com/dir/dir2/dir3']
>>> # as you see above, now the results are correct
...

Work environment

  • Client OS: RHEL
  • Server OS: RHEL
  • Browse type and version: n/a
  • Cortex version: 3.1
  • Cortex Analyzer/Responder name: EmlParser
  • Cortex Analyzer/Responder version: EmlParser version 1.2; repo git tag 2.9.4

Possible solutions
Instead of passing the whole dictionary as a string, we can iterate over the values of the dictionary. Just like shown above.
The patch that does this job, and is verified to be working:
iocextract_url_parsing_fix.txt

Additional context
Add any other context about the problem here.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the EmlParser artifacts() method and compare its current iocextract.extract_urls() input with the provided Python examples. Verify that parsing dictionary values avoids escaped carriage-return and newline text, then reproduce the sample URL extraction to confirm the resulting URLs end before those sequences.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
security
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.