apache / apache/druid

Broker fails query with `JsonParseException: Invalid type marker byte 0x3c` when Historical/Broker returns Jetty's HTML error page for HTTP 429 / 503

Open
#19,407 0 comments 0 reactions 0 assignees View on GitHub
Uncategorized problem report
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

Apache Druid 36.0.0

### Description

When a Druid data server (Historical or Broker) is under load and rejects a `/druid/v2/` POST with HTTP 429 (via `LimitRequestsFilter`) or HTTP 503 (via Jetty's `QoSFilter` suspend timeout), the response body is **HTML**, not JSON, because Druid relies on Jetty's default `ErrorHandler`. The upstream Druid Broker/Router does not check `Content-Type` before parsing, so Jackson fails on the very first byte (`0x3c` = `<`), and the user-visible error is a generic `Unknown exception` HTTP 500 instead of a meaningful `Too Many Requests` / capacity-exceeded error.

#### Cluster size

Reproduced on a single-server `micro-quickstart` deployment (1× Coordinator+Overlord, 1× Broker, 1× Historical, 1× Router, 1× MiddleManager).

#### Configurations in use

Historical `runtime.properties` (intentionally constrained to force saturation; production-realistic values would be larger but the same code path triggers under sufficient load):

```properties
druid.server.http.numThreads=4
druid.server.http.enableRequestLimit=true
druid.server.http.maxIdleTime=PT5S
druid.processing.numThreads=1
druid.processing.numMergeBuffers=2
druid.processing.buffer.sizeBytes=64MiB
```

Broker `runtime.properties`:

```properties
druid.broker.http.numConnections=20
druid.broker.http.readTimeout=PT2M
druid.broker.http.unusedConnectionTimeout=PT4S
druid.server.http.numThreads=20
```

Debug logging on Broker / Historical:
- `org.apache.druid.jetty.RequestLog=DEBUG`
- `org.apache.druid.client.DirectDruidClient=DEBUG`
- `org.apache.druid.client.JsonParserIterator=DEBUG`
- `org.eclipse.jetty.server.handler.ErrorHandler=DEBUG`
- `org.eclipse.jetty.util.thread.QueuedThreadPool=DEBUG`

#### Steps to reproduce

1. Apply the constrained Historical config above and restart the Historical.
2. Drive a concurrent load against the Router using a small set of queries on the `wikipedia` datasource (e.g., 1000 requests at concurrency 200).
3. Observe HTTP 500 responses on the client containing `JsonParseException`, and matching HTTP 429 in the Historical request log.

A direct probe against the Historical (or any inspection of `historical.log`) shows the HTML body:

```
HTTP/1.1 429 Too Many Requests
Content-Type: text/html;charset=ISO-8859-1

Error 429 Too Many Requests

HTTP ERROR 429 Too Many Requests


...

```

#### Error message / stack trace

Broker:

```
WARN [ForkJoinPool-1-worker-3] org.apache.druid.client.JsonParserIterator -
Query [5710bfb8-ade6-4d13-8211-641c7e2f512f] to host [localhost:8083] interrupted

com.fasterxml.jackson.core.JsonParseException: Invalid type marker byte 0x3c
for expected value token
at [Source: (SequenceInputStream); byte offset: #1]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:724)
...
at org.apache.druid.client.JsonParserIterator.init(JsonParserIterator.java:216)
at org.apache.druid.client.DirectDruidClient.run(DirectDruidClient.java:381)
```

Client (through Router):

```json
HTTP/1.1 500 Internal Server Error
{"error":"Unknown exception",
"errorClass":"com.fasterxml.jackson.core.JsonParseException",
"host":"localhost:8083",
"errorCode":"legacyQueryException",
"persona":"OPERATOR",
"category":"RUNTIME_FAILURE"}
```

#### Debugging done

Cross-checked against Druid sources:

- `server/src/main/java/org/apache/druid/server/initialization/jetty/LimitRequestsFilter.java`
emits `((HttpServletResponse) response).sendError(429, "Too Many Requests")`,
which dispatches to Jetty's default `ErrorHandler` — producing HTML.
- `server/src/main/java/org/apache/druid/server/initialization/jetty/JettyServerModule.java` (around line 469)
only customizes the `ErrorHandler` when `druid.server.http.showDetailedJettyErrors=false`,
and even then it merely clears `RequestDispatcher.ERROR_EXCEPTION` and delegates to
`super.handle(...)`. The body rendering remains Jetty's default (HTML when no
`Accept: application/json` header is sent, which is the case for `DirectDruidClient`).
- `server/src/main/java/org/apache/druid/client/DirectDruidClient.java` does not check
`Content-Type`. The `InputStream` is handed straight to
`JsonParserIterator.init()` (`server/src/main/java/org/apache/druid/client/JsonParserIterator.java`),
which calls `objectMapper.getFactory().createParser(is); jp.nextToken();` —
exactly where Jackson trips over the leading `<`.
- The `QoSFilter` registered in
`services/src/main/java/org/apache/druid/cli/QueryJettyServerInitializer.java`
produces HTTP 503 on suspend timeout via the same code path, so 503 responses can
trigger the identical failure.
- `docs/configuration/index.md` description of `druid.server.http.showDetailedJettyErrors`
("the JSON response only includes...") is misleading: the default error handler still
performs `Accept`-header based content negotiation and falls back to HTML for clients
that do not advertise `application/json`.

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the constrained Historical settings and concurrent Router load, then read DirectDruidClient.java and JsonParserIterator.java alongside LimitRequestsFilter.java, JettyServerModule.java, and QueryJettyServerInitializer.java. Verify how HTML 429/503 responses are handled, and define done as preventing the Jackson parse failure while returning a meaningful capacity or rate-limit error to the client.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.