TheHive-Project / TheHive-Project/Cortex

Fix broken compatibility with Elasticsearch 8.x/Opensearch 2.x

Open
#429 7 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Scala
Stars
1.6k
Forks
264
PR merge metrics
No merged PRs in 30d

Description

Request Type

Bug

Work Environment
Question Answer
OS version (server) Ubuntu
OS version (client) 20.04
Cortex version / git hash 3.1.6-withdeps
Package Type Docker
Browser type & version N/A
Problem Description

Hi, I identified problem with creating cortex database in Elasticsearch >=8.x and OpenSearch >= 2.x. Parameter include_type_name is removed from newest ES/OS versions (reference Moving from types to typeless APIs in Elasticsearch 7.0 for ES, and Remove mapping types #150 for OS).

Steps to Reproduce
  1. Setup Opensearch 2.x or ElasticSearch 8.x.
  2. Run clean cortex install.
  3. Wait for cortex to setup, and click migrate database button.
  4. See error message in logs.
Possible Solutions

Cortex uses elastic4play library, which uses elastic4s under the hood. Bumping elastic4s version from 7.17.2 to 8.x should enable compatibility with new Elasticsearch/Opensearch versions.

Complementary information
[error] o.e.d.DBConfiguration - ElasticSearch request failure: PUT:/cortex_6?include_type_name=false
StringEntity({"settings":{"index":{"number_of_shards":5,"number_of_replicas":1,"mapping.nested_fields.limit":100}},"mappings":{"date_detection":false,"numeric_detection":false,
...
"job":["dummy-job","report"],"sequence":["dummy-sequence"],"report":["artifact"],"audit":["dummy-audit"],"user":["dummy-user"],"dblist":["dummy-dblist"]}}}}},Some(application/json))
 => ElasticError(illegal_argument_exception,request [/cortex_6] contains unrecognized parameter: [include_type_name],None,None,None,List(ElasticError(illegal_argument_exception,request [/cortex_6] contains unrecognized parameter: [include_type_name],None,None,None,null,None,None,None,List())),None,None,None,List())

https://github.com/TheHive-Project/elastic4play/blob/86665bfe13a5cb34104482ebe49039d309f23f43/build.sbt#L46

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the elastic4s version declaration linked in build.sbt and trace the database setup path that produces the PUT request shown in the logs. Reproduce the migration against Elasticsearch 8.x or OpenSearch 2.x, then verify that a clean Cortex installation completes database creation without the unsupported include_type_name parameter.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, scala
Domain
backend, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.