TheHive-Project / TheHive-Project/Cortex

issue running cortexneurons via docker

Open
#476 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Scala
Stars
1.6k
Forks
264
PR merge metrics
No merged PRs in 30d

Description

Request Type

Bug

Work Environment
Question Answer
OS version (server) Ubuntu

| Cortex version / git hash | docker image: 'thehiveproject/cortex:3.1.1-1'
| Package Type | Docker

Problem Description

I've setup cortex on a dedicated server.
Cortex run as a docker using docker compose, using instruction at
https://docs.strangebee.com/cortex/installation-and-configuration/run-cortex-with-docker/#use-docker-compose

yml file is the following:

version: "2"
services:
elasticsearch:
image: elasticsearch:7.11.1
environment:
- http.host=0.0.0.0
- discovery.type=single-node
- script.allowed_types=inline
- thread_pool.search.queue_size=100000
- thread_pool.write.queue_size=10000
cortex:
image: 'thehiveproject/cortex:3.1.1-1'
environment:
- 'job_directory=${job_directory}'
volumes:
- '/var/run/docker.sock:/var/run/docker.sock'
- '${job_directory}:${job_directory}'
depends_on:
- elasticsearch
ports:
- '0.0.0.0:9001:9001'

I can start the service with docker-compose up, I can finalize the configuration, setup few analyzers with required API
but when I try to run any of the analyzers I get the following error:

Request error: POST unix://localhost:80/images/create?fromImage=docker.io%2Fcortexneurons%2Fvirustotal_getreport&tag=3: 500, body: {"message":"toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"}

or

Request error: POST unix://localhost:80/images/create?fromImage=docker.io%2Fcortexneurons%2Fabuseipdb&tag=1: 500, body: {"message":"toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"}

even if I successfully manually login to docker.com with the proper user/token the error persist, as if the further download of the cortexneurons (initial pull) is not using such authentication and fails because of rate-limit on the docker.com side.

  • Sometimes, when I do not hit the rate limit threshold, the analyzer works as expected.

  • Is there a way to configure authentication parameters to be used whenever a pull is required?

Log when error occurs:
cortex-1 | [info] o.t.c.s.JobSrv - Looking for similar job in the last 10 minutes (worker=3e67614b9a076dba5ed4aa7fd53c75b7, dataType=ip, data=Left(8.8.8.8), tlp=2, parameters={})
cortex-1 | [info] o.t.c.s.AuditActor - Job 0e6X_JIBzEDw4VX1_5dT has be updated (JsDefined("InProgress"))
cortex-1 | [info] o.t.c.s.AuditActor - Job 0e6X_JIBzEDw4VX1_5dT has be updated (JsDefined("Failure"))
cortex-1 | [info] o.t.c.s.JobSrv - Job 0e6X_JIBzEDw4VX1_5dT has finished with status Failure

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked Docker Compose installation instructions and the supplied compose YAML, especially the Cortex container's Docker socket and analyzer image pulls. Reproduce an analyzer run after Docker authentication, then determine whether authenticated credentials are passed to the daemon; done means analyzer images can be pulled without Docker Hub rate-limit failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.