Some Spark jobs generate com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60))

Nobody has claimed this yet.

Latest 2.0.6 Dr Elephant with the new Spark fetchers throws this in SparkRestClient.scala:

com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')

seemingly when the Spark job name contains a space. This is happening on every Spark job I have with a space in the name, but works fine when there is no space in the name.

EXAMPLE:

When running 2.0.6, this Spark REST API call:

http://scrubbed.url.cardlytics.com:18080/api/v1/applications/application_1475605876965_0216

returns this to a browser:

{
"id": "application_1475605876965_0216",
"name": "partnerClean 20160803", <--- one space between partnerClean and the yyyymmdd value
"attempts": [
{
"attemptId": "2",
"startTime": "2016-10-05T15:50:57.802GMT",
"endTime": "2016-10-05T16:17:01.937GMT",
"sparkUser": "jobuser",
"completed": true
}
]
}

but Dr Elephant blows up:

12-22-2016 14:43:27 INFO [dr-el-executor-thread-0] com.linkedin.drelephant.ElephantRunner : Analyzing SPARK application_1475605876965_0216
12-22-2016 14:43:27 INFO [dr-el-executor-thread-0] com.linkedin.drelephant.spark.fetchers.SparkFetcher : Fetching data for application_1475605876965_0216
12-22-2016 14:43:27 INFO [ForkJoinPool-1-worker-55] com.linkedin.drelephant.spark.fetchers.SparkRestClient : calling REST API at http://scrubbed.url.cardlytics.com:18080/api/v1/applications/application_1475605876965_0216
12-22-2016 14:43:27 ERROR [ForkJoinPool-1-worker-55] com.linkedin.drelephant.spark.fetchers.SparkRestClient : error reading ApplicationInfo from http://scrubbed.url.cardlytics.com:18080/api/v1/applications/application_1475605876965_0216
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source:


History Server

; line: 1, column: 2]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1487)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:518)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:447)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1461)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:683)
at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3602)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3547)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2616)
at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper$class.readValue(ScalaObjectMapper.scala:184)
at com.linkedin.drelephant.spark.fetchers.SparkRestClient$$anon$1.readValue(SparkRestClient.scala:173)
at com.linkedin.drelephant.spark.fetchers.SparkRestClient$$anonfun$getApplicationInfo$1.apply(SparkRestClient.scala:115)
at com.linkedin.drelephant.spark.fetchers.SparkRestClient$$anonfun$getApplicationInfo$1.apply(SparkRestClient.scala:115)
at com.linkedin.drelephant.spark.fetchers.SparkRestClient$.get(SparkRestClient.scala:180)
at com.linkedin.drelephant.spark.fetchers.SparkRestClient.getApplicationInfo(SparkRestClient.scala:115)
at com.linkedin.drelephant.spark.fetchers.SparkRestClient.fetchData(SparkRestClient.scala:73)
at com.linkedin.drelephant.spark.fetchers.SparkFetcher$stateMachine$1$1.apply(SparkFetcher.scala:90)
at com.linkedin.drelephant.spark.fetchers.SparkFetcher$stateMachine$1$1.apply$mcV$sp(SparkFetcher.scala:89)
at com.linkedin.drelephant.spark.fetchers.SparkFetcher$stateMachine$1$1.apply(SparkFetcher.scala:89)
at com.linkedin.drelephant.spark.fetchers.SparkFetcher$stateMachine$1$1.apply(SparkFetcher.scala:89)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at scala.concurrent.impl.ExecutionContextImpl$$anon$3.exec(ExecutionContextImpl.scala:107)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Another example:

http://scrubbed.url.cardlytics.com:18080/api/v1/applications/application_1475605876965_0256/1/jobs

returns to the browser:

[
{
"jobId": 6,
"name": "json at NativeMethodAccessorImpl.java:-2",
"submissionTime": "2016-10-05T23:09:06.755GMT",
"completionTime": "2016-10-05T23:09:07.861GMT",
"stageIds": [
16,
17
],
"status": "SUCCEEDED",
"numTasks": 248,
"numActiveTasks": 0,
"numCompletedTasks": 200,
"numSkippedTasks": 200,
"numFailedTasks": 0,
"numActiveStages": 0,
"numCompletedStages": 1,
"numSkippedStages": 1,
"numFailedStages": 0
},
{
"jobId": 5,
"name": "saveAsTextFile at package.scala:209",
"submissionTime": "2016-10-05T23:09:01.311GMT",
"completionTime": "2016-10-05T23:09:05.678GMT",
"stageIds": [
15,
14
],
"status": "SUCCEEDED",
"numTasks": 248,
"numActiveTasks": 0,
"numCompletedTasks": 200,
"numSkippedTasks": 200,
"numFailedTasks": 0,
"numActiveStages": 0,
"numCompletedStages": 1,
"numSkippedStages": 1,
"numFailedStages": 0
},
{
"jobId": 4,
"name": "count at NativeMethodAccessorImpl.java:-2",
"submissionTime": "2016-10-05T23:09:00.227GMT",
"completionTime": "2016-10-05T23:09:00.935GMT",
"stageIds": [
12,
13,
11
],
"status": "SUCCEEDED",
"numTasks": 249,
"numActiveTasks": 0,
"numCompletedTasks": 201,
"numSkippedTasks": 201,
"numFailedTasks": 0,
"numActiveStages": 0,
"numCompletedStages": 2,
"numSkippedStages": 1,
"numFailedStages": 0
},
{
"jobId": 3,
"name": "count at NativeMethodAccessorImpl.java:-2",
"submissionTime": "2016-10-05T23:08:59.097GMT",
"completionTime": "2016-10-05T23:09:00.149GMT",
"stageIds": [
9,
10,
8
],
"status": "SUCCEEDED",
"numTasks": 249,
"numActiveTasks": 0,
"numCompletedTasks": 201,
"numSkippedTasks": 201,
"numFailedTasks": 0,
"numActiveStages": 0,
"numCompletedStages": 2,
"numSkippedStages": 1,
"numFailedStages": 0
},
{
"jobId": 2,
"name": "count at NativeMethodAccessorImpl.java:-2",
"submissionTime": "2016-10-05T23:08:58.542GMT",
"completionTime": "2016-10-05T23:08:58.992GMT",
"stageIds": [
5,
6,
7
],
"status": "SUCCEEDED",
"numTasks": 249,
"numActiveTasks": 0,
"numCompletedTasks": 201,
"numSkippedTasks": 201,
"numFailedTasks": 0,
"numActiveStages": 0,
"numCompletedStages": 2,
"numSkippedStages": 1,
"numFailedStages": 0
},
{
"jobId": 1,
"name": "count at NativeMethodAccessorImpl.java:-2",
"submissionTime": "2016-10-05T22:56:00.675GMT",
"completionTime": "2016-10-05T23:08:58.157GMT",
"stageIds": [
2,
3,
4
],
"status": "SUCCEEDED",
"numTasks": 249,
"numActiveTasks": 0,
"numCompletedTasks": 249,
"numSkippedTasks": 249,
"numFailedTasks": 0,
"numActiveStages": 0,
"numCompletedStages": 3,
"numSkippedStages": 0,
"numFailedStages": 0
},
{
"jobId": 0,
"name": "json at NativeMethodAccessorImpl.java:-2",
"submissionTime": "2016-10-05T22:31:42.545GMT",
"completionTime": "2016-10-05T22:55:58.933GMT",
"stageIds": [
0,
1
],
"status": "SUCCEEDED",
"numTasks": 54,
"numActiveTasks": 0,
"numCompletedTasks": 54,
"numSkippedTasks": 54,
"numFailedTasks": 0,
"numActiveStages": 0,
"numCompletedStages": 2,
"numSkippedStages": 0,
"numFailedStages": 0
}
]

but Dr Elephant throws the same error.

Start in SparkRestClient.scala and inspect how the Spark REST API response is requested and parsed for application IDs whose job names contain spaces. Reproduce the reported application request and compare the returned HTML with the expected JSON; done means affected Spark jobs are handled without the reported JsonParseException.

Written by the indexing model from the issue text.