TheHive-Project / TheHive-Project/Cortex

The analyzers associated with file analysis stopped working. Need some help

Open
#350 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Scala
Stars
1.6k
Forks
264
PR merge metrics
No merged PRs in 30d

Description

Describe the bug
The analyzers associated with file analysis stopped working.

To Reproduce
The reason is not clear.

Complementary information
When trying to run File_info_7 to any file. Cortex log:

> 2021-03-04 06:18:48,179 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57220 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0     
> ByteArrayEntity([B@5e9c4c6a,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:48,179 [WARN] from org.elastic4play.utils.Retry in application-akka.actor.default-dispatcher-57220 - An error occurs (data 09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0 not found), retrying (5)
> 2021-03-04 06:18:48,180 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57226 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1     
> ByteArrayEntity([B@70e6294f,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:49,204 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57226 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1     
> ByteArrayEntity([B@3f4a502a,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:49,204 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57225 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0     
> ByteArrayEntity([B@1d9cfe18,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:49,204 [WARN] from org.elastic4play.utils.Retry in application-akka.actor.default-dispatcher-57225 - An error occurs (data 09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0 not found), retrying (4)
> 2021-03-04 06:18:51,223 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57225 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0     
> ByteArrayEntity([B@36976aa1,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:51,223 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57223 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1     
> ByteArrayEntity([B@5bf72133,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:51,223 [WARN] from org.elastic4play.utils.Retry in application-akka.actor.default-dispatcher-57223 - An error occurs (data 09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0 not found), retrying (3)
> 2021-03-04 06:18:55,244 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57220 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0     
> ByteArrayEntity([B@5464271a,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:55,244 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57222 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1     
> ByteArrayEntity([B@5b62b771,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:55,244 [WARN] from org.elastic4play.utils.Retry in application-akka.actor.default-dispatcher-57220 - An error occurs (data 09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0 not found), retrying (2)
> 2021-03-04 06:19:03,264 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57220 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0     
> ByteArrayEntity([B@1166b3f2,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:19:03,264 [WARN] from org.elastic4play.utils.Retry in application-akka.actor.default-dispatcher-57220 - An error occurs (data 09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0 not found), retrying (1)
> 2021-03-04 06:19:03,264 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57220 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1
> ByteArrayEntity([B@2d2542a8,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:19:19,284 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57220 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1
> ByteArrayEntity([B@7507514d,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:19:19,284 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57222 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0
> ByteArrayEntity([B@519adc01,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:19:19,285 [INFO] from org.thp.cortex.services.ErrorHandler in application-akka.actor.default-dispatcher-57226 - POST /api/analyzer/b339e64bd52c52bfceb56550e5f7bf31/run returned 404
> org.elastic4play.NotFoundError: data 09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0 not found
>       at org.elastic4play.database.DBGet.$anonfun$apply$2(DBGet.scala:33)
>       at scala.Option.fold(Option.scala:251)
>       at org.elastic4play.database.DBGet.$anonfun$apply$1(DBGet.scala:33)
>       at scala.util.Success.$anonfun$map$1(Try.scala:255)
>       at scala.util.Success.map(Try.scala:213)
>       at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
>       at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
>       at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
>       at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
>       at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
>       at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
>       at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
>       at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
>       at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)
>       at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
>       at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
>       at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
>       at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
>       at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
>       at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

Work environment

  • Server OS: Ubuntu 20.04.1

  • Browse type and version:

  • Cortex Analyzer FileInfo_7_0

  • Cortex: 3.1.0-1

  • Elastic4Play: 1.12.3

  • Play: 2.8.3

  • Elastic4s: 7.9.1

  • ElasticSearch client: 7.9.1

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing a File_info_7_0 run through POST /api/analyzer/{id}/run and review the Cortex and Elasticsearch errors shown in the issue. Trace the failed document creation and the reported unknown join name [data] for field [relations]. Done means file analysis completes without the 404 and the analyzer result is stored successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch
Domain
api, backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.