Proposal: optimize job execution

Request Type

Feature Request

Problem Description

Cortex is amazing TI platform, but it's hard to use for batch observable analysis or responding, because underneath it uses exec and spawns a process for every new job, which is mostly unnecessary expensive, because the execution is I/O bound. If I want to get relatively cheap MaxMind geolocation for 100 IPs or send email abuses for 500 domains, it won't be an easy task for Cortex.

Possible Solutions

The current workaround that I'm aware of is to add a new observable data type, which can hold multiple lines of indicators. And we actually use it, but it lacks clean and descriptive taxonomies - just imagine taxonomies for 100 indicators; and proper error handling - what if the responding fails at some entry?

The proposal is to change the job starting scheme from just feeding file execution stdin with json to pushing tasks and receiving results from some message bus or a queue, making the whole execution process more lightweight and effective.

The job receiver and results pusher component should be written in the same language as analyzers and responders, because it will invoke them as threads or coroutines (therefore python), but it would be great if the implementation won't be tighten to just a single language. To achieve this any open message queue can be used: RabbitMQ, NATS, Kafka, NSQ or even Redis.

I think it's possible to make a smooth migration without rewriting any analyzer or responder at all by making changes just to cortexutils after the message queue will be chosen.

I understand that it's not a simple change and won't gonna happen tomorrow or in the near future, but I think it would be great to consider for the new major version at least. I will gladly help with what I can.

Thank you for your enormous work guys!