TheHive-Project / TheHive-Project/Cortex-Analyzers

[Bug] The analyzers associated with file analysis stopped working. Need some help.

Open
#956 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
490
Forks
405
Avg merge
2d 43m
Merged PRs (30d)
8

Description

Describe the bug
The analyzers associated with file analysis stopped working.

To Reproduce
The reason is not clear.

Complementary information
When trying to run File_info_7 to any file. Cortex log:

> 2021-03-04 06:18:48,179 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57220 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0     
> ByteArrayEntity([B@5e9c4c6a,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:48,179 [WARN] from org.elastic4play.utils.Retry in application-akka.actor.default-dispatcher-57220 - An error occurs (data 09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0 not found), retrying (5)
> 2021-03-04 06:18:48,180 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57226 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1     
> ByteArrayEntity([B@70e6294f,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:49,204 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57226 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1     
> ByteArrayEntity([B@3f4a502a,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:49,204 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57225 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0     
> ByteArrayEntity([B@1d9cfe18,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:49,204 [WARN] from org.elastic4play.utils.Retry in application-akka.actor.default-dispatcher-57225 - An error occurs (data 09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0 not found), retrying (4)
> 2021-03-04 06:18:51,223 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57225 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0     
> ByteArrayEntity([B@36976aa1,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:51,223 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57223 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1     
> ByteArrayEntity([B@5bf72133,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:51,223 [WARN] from org.elastic4play.utils.Retry in application-akka.actor.default-dispatcher-57223 - An error occurs (data 09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0 not found), retrying (3)
> 2021-03-04 06:18:55,244 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57220 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0     
> ByteArrayEntity([B@5464271a,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:55,244 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57222 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1     
> ByteArrayEntity([B@5b62b771,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:55,244 [WARN] from org.elastic4play.utils.Retry in application-akka.actor.default-dispatcher-57220 - An error occurs (data 09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0 not found), retrying (2)
> 2021-03-04 06:19:03,264 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57220 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0     
> ByteArrayEntity([B@1166b3f2,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:19:03,264 [WARN] from org.elastic4play.utils.Retry in application-akka.actor.default-dispatcher-57220 - An error occurs (data 09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0 not found), retrying (1)
> 2021-03-04 06:19:03,264 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57220 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1
> ByteArrayEntity([B@2d2542a8,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:19:19,284 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57220 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1
> ByteArrayEntity([B@7507514d,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:19:19,284 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57222 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0
> ByteArrayEntity([B@519adc01,Some(application/json))
>  => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:19:19,285 [INFO] from org.thp.cortex.services.ErrorHandler in application-akka.actor.default-dispatcher-57226 - POST /api/analyzer/b339e64bd52c52bfceb56550e5f7bf31/run returned 404
> org.elastic4play.NotFoundError: data 09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0 not found
>       at org.elastic4play.database.DBGet.$anonfun$apply$2(DBGet.scala:33)
>       at scala.Option.fold(Option.scala:251)
>       at org.elastic4play.database.DBGet.$anonfun$apply$1(DBGet.scala:33)
>       at scala.util.Success.$anonfun$map$1(Try.scala:255)
>       at scala.util.Success.map(Try.scala:213)
>       at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
>       at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
>       at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
>       at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
>       at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
>       at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
>       at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
>       at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
>       at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)
>       at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
>       at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
>       at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
>       at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
>       at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
>       at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

Work environment

  • Server OS: Ubuntu 20.04.1

  • Browse type and version:

  • Cortex Analyzer FileInfo_7_0

  • Cortex: 3.1.0-1

  • Elastic4Play: 1.12.3

  • Play: 2.8.3

  • Elastic4s: 7.9.1

  • ElasticSearch client: 7.9.1

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with the Cortex Analyzer FileInfo_7_0 and inspect the logged Elasticsearch request for the unknown join name [data] in field [relations]. Trace the analyzer run through POST /api/analyzer/{id}/run and the related data persistence path. Done means file analysis completes without the Elasticsearch parsing error or resulting 404.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, python
Domain
backend, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.