[FR] Decentralize Docker Build Process and move towards complete docker based analyzers / responders / modules

Feature description
Decentralize Docker Build Process and move towards complete docker based analyzers / responders / modules.

Describe the solution you'd like

This is somewhat related to https://github.com/TheHive-Project/Cortex-Analyzers/pull/607

@nadouani @milesflo I have a different take to this. And below is just me thinking out loud and putting my ideas.

Dockerizing all analyzers is a great idea. In fact I would suggest one step ahead on dockerizing all extensible modules of Cortex including responders.

But it being done in the set of huge number of commits (as part of mentioned PR) with varied changes is not a great idea, though I appreciate the work that went behind correcting this. This is not a scalable approach, as whenever something breaks due to centralised docker image building logic with a given analyzer as proposed in this PR, every analyzer's docker image and central docker image building code has to be touched which would be an absolute headache and it would be chaotic. Decentralise all these.

Dockerize Everything. Decentralize everything !!

The approach we can take is , "Bring your own docker image" . This will avoid headache of maintaing and touching multiple analyzers to fix one issue , which has been the case till date. And its not a scalable method for sure.I would also suggest we move away from bias of python where we honour the requirements.txt in case of python alone. Lets have a generalised approach where cortex core is different and all that it knows is to build a docker image (in case of private image) , execute a docker image (any module it may be - analyzer / responder / any module) with a set of arguments either as environment variable passed on to the docker container or command arguments during runtime. This makes it much more modular.

Centralised build logic would fail unless we have a highly opinionated framework on what can be used and what not but its difficult to have such an opinionated framework which can cover all the possible use cases for an analyzer at the same time stick to the strict standard of the Cortex Framework. Also when it comes to language agnostic strategy, such a framework is not possible to be built covering all possible languages with all possible use cases. Hence this is not a viable solution in my humble opinion.

Also I actually don't see any value / significance of having a centralised build logic if all are going to be in their own bubble with complete isolation and operating as a separate computing entity. If security is a concern, one can run a docker scanner like trivy to iron out the security vulnerablities during submission and during every periodic sanity check.

Language Agnostic and Clean management of Core

This has an advantage where cortex core need not have to bother about a selective approach towards a specific progamming language and running logics. All that it has to care is to how to run the module using docker runtime and get the exit status of it to report on Pass or Failure of a Job and get the output of it (in this case the enrichment info) and show it in UI. So all cortex has to care about is what goes into docker , docker container's runtime exit status (for reporting failure) , and what comes out of docker container run.

Components to be submitted by a developer or contributor as part of an Analyzer (or a module) submission PR

Every developer who contributes the analyzer / responder / any extensible cortex module that may come in future should come up with the following three things :-

i. Dockerfile customised for her / his specific module (analyzer / responder / any extensible cortex module that may come in future)

As mentioned earlier, its better we follow "Bring your own Docker Image". We should not interfere in the build logic of analyzer nor the dockerfile. It should be totally decentralised. Its the responsibility of the developer who build the analyzer, as developer of analyzer knows better (Ideal Assumption) what needs to be setup for her / his analyzer / responder and how to build the right docker image for that module to work properly. Let the contributor decide the build logic, the components and required packages and the maintainers can verify it.This is where decentralisation proves to be advantage over central building logic.

ii. The code folder containing all the logic of processing. (Any Language) - So that cortex can build up all the analysers docker image during initialisation and keep it ready for use later (Or can pull from a container registry whichever works / chosen by user). Image caching logic can be thought later.This can be extended to if analyzer needs to be updated, we can have an option in ui for force update or some cron job to check the git pull status on a periodic basis from cortex and monitor the changes to analyzers , and cortex can pull the code and rebuild it / pull the image from registry.

iii. Service Interaction File as usual .

Automated Basic Accepting Criteria to reduce the team's time on review of PR

As part of accepting the PR , one of the criteria to accept the contribution or test of pass is , we build the provided Docker Image submitted and if the docker builds successfully and takes in an argument and provides an expected successful output, it is functionally good and is fit to go to next stage of review. If not , reject the PR. Also one can run couple of docker (Dockerfile lint + Docker Image Scan) scan for identifying bad practice / vulnerable version and can consider that as well as one of the criteria for moving to next stage.

As part of finding the bad modules / analyzers / extensible modules. We can build the docker images on a fortnightly basis (or some periodicity) and run the functional test of pass to check if the analyzer is still functional or if there is any exit 1 status encountered or any undesired output received and docker security checks (Docker Image Scan) , this can be further used to notify the contributors to check the analyzer again and then give a set of time to fix or decommision it from official repo . This will be helpful in maintaining the sanity of analyzers / responders and have a clean working set of modules in the official repo.

Summary
Long story short :-
i. Dockerize all . Leave the language bias
ii. Decentralize the build logic of the modules.
iii. Provide Freedom to Developers of the module (an extension of iii). Leave the responsibility of module working to the individual contributor with retention of control over to whether to continue include as part of official repository with the owner of the repository.
iv. Have basic automated acceptance criteria to filter the PRs to be reviewed by the team.
v. Periodic checking of Analyzers in CD to maintain sanity of the analysers / modules repository.

This entire proposal is a complete project of its own and may require some good amount of time.The suggested ideas may have flaws, feel free to correct them, but its a step towards modularity. I have built an internal framework along with my team for a different purpose with the same idea (and other more fitting our context) suggested here by me. This is a proven working model for us.The context of our application is for different security purpose though. I don't know how it may span out for Cortex. But just by logical reasoning I have a fair confidence it should roll out good.

I know some of the above has been implemented in some form, but I chose to put my entire idea anyway so that a flow can be properly projected.

Feel free to ignore if you dont find any value in this entire piece :P . Just thought of putting my idea for greater benefit.

Additional context
None.

The issue names no files, tests, or entry points; begin by tracing the existing centralized Docker build and module execution flow in Cortex-Analyzers. Define the Dockerfile, module code, service interaction, build and acceptance checks, and periodic validation required for a complete proposal, with a successful build and expected output as the stated done criteria.