TheHive-Project / TheHive-Project/Cortex
The analyzers associated with file analysis stopped working. Need some help
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 264
- PR merge metrics
- No merged PRs in 30d
Description
Describe the bug
The analyzers associated with file analysis stopped working.
To Reproduce
The reason is not clear.
Complementary information
When trying to run File_info_7 to any file. Cortex log:
> 2021-03-04 06:18:48,179 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57220 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0
> ByteArrayEntity([B@5e9c4c6a,Some(application/json))
> => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:48,179 [WARN] from org.elastic4play.utils.Retry in application-akka.actor.default-dispatcher-57220 - An error occurs (data 09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0 not found), retrying (5)
> 2021-03-04 06:18:48,180 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57226 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1
> ByteArrayEntity([B@70e6294f,Some(application/json))
> => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:49,204 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57226 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1
> ByteArrayEntity([B@3f4a502a,Some(application/json))
> => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:49,204 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57225 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0
> ByteArrayEntity([B@1d9cfe18,Some(application/json))
> => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:49,204 [WARN] from org.elastic4play.utils.Retry in application-akka.actor.default-dispatcher-57225 - An error occurs (data 09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0 not found), retrying (4)
> 2021-03-04 06:18:51,223 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57225 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0
> ByteArrayEntity([B@36976aa1,Some(application/json))
> => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:51,223 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57223 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1
> ByteArrayEntity([B@5bf72133,Some(application/json))
> => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:51,223 [WARN] from org.elastic4play.utils.Retry in application-akka.actor.default-dispatcher-57223 - An error occurs (data 09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0 not found), retrying (3)
> 2021-03-04 06:18:55,244 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57220 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0
> ByteArrayEntity([B@5464271a,Some(application/json))
> => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:55,244 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57222 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1
> ByteArrayEntity([B@5b62b771,Some(application/json))
> => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:18:55,244 [WARN] from org.elastic4play.utils.Retry in application-akka.actor.default-dispatcher-57220 - An error occurs (data 09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0 not found), retrying (2)
> 2021-03-04 06:19:03,264 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57220 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0
> ByteArrayEntity([B@1166b3f2,Some(application/json))
> => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:19:03,264 [WARN] from org.elastic4play.utils.Retry in application-akka.actor.default-dispatcher-57220 - An error occurs (data 09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0 not found), retrying (1)
> 2021-03-04 06:19:03,264 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57220 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1
> ByteArrayEntity([B@2d2542a8,Some(application/json))
> => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:19:19,284 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57220 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_1
> ByteArrayEntity([B@7507514d,Some(application/json))
> => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:19:19,284 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-57222 - ElasticSearch request failure: PUT:/cortex_5/_doc/09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0?refresh=wait_for&op_type=create&routing=09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0
> ByteArrayEntity([B@519adc01,Some(application/json))
> => ElasticError(mapper_parsing_exception,failed to parse,None,None,None,List(ElasticError(mapper_parsing_exception,failed to parse,None,None,None,null,None,None,None,List())),Some(CausedBy(illegal_argument_exception,unknown join name [data] for field [relations],Map())),None,None,List())
> 2021-03-04 06:19:19,285 [INFO] from org.thp.cortex.services.ErrorHandler in application-akka.actor.default-dispatcher-57226 - POST /api/analyzer/b339e64bd52c52bfceb56550e5f7bf31/run returned 404
> org.elastic4play.NotFoundError: data 09bb2cfb2684fc862b480b75f051e2fe50de9d5d324f54e53dff48503cf229c0_0 not found
> at org.elastic4play.database.DBGet.$anonfun$apply$2(DBGet.scala:33)
> at scala.Option.fold(Option.scala:251)
> at org.elastic4play.database.DBGet.$anonfun$apply$1(DBGet.scala:33)
> at scala.util.Success.$anonfun$map$1(Try.scala:255)
> at scala.util.Success.map(Try.scala:213)
> at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
> at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
> at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
> at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
> at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
> at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
> at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Work environment
-
Server OS: Ubuntu 20.04.1
-
Browse type and version:
-
Cortex Analyzer FileInfo_7_0
-
Cortex: 3.1.0-1
-
Elastic4Play: 1.12.3
-
Play: 2.8.3
-
Elastic4s: 7.9.1
-
ElasticSearch client: 7.9.1
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing a File_info_7_0 run through POST /api/analyzer/{id}/run and review the Cortex and Elasticsearch errors shown in the issue. Trace the failed document creation and the reported unknown join name [data] for field [relations]. Done means file analysis completes without the 404 and the analyzer result is stored successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch
- Domain
- api, backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100