Broker fails query with `JsonParseException: Invalid type marker byte 0x3c` when Historical/Broker returns Jetty's HTML error page for HTTP 429 / 503
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
Apache Druid 36.0.0
### Description
When a Druid data server (Historical or Broker) is under load and rejects a `/druid/v2/` POST with HTTP 429 (via `LimitRequestsFilter`) or HTTP 503 (via Jetty's `QoSFilter` suspend timeout), the response body is **HTML**, not JSON, because Druid relies on Jetty's default `ErrorHandler`. The upstream Druid Broker/Router does not check `Content-Type` before parsing, so Jackson fails on the very first byte (`0x3c` = `<`), and the user-visible error is a generic `Unknown exception` HTTP 500 instead of a meaningful `Too Many Requests` / capacity-exceeded error.
#### Cluster size
Reproduced on a single-server `micro-quickstart` deployment (1× Coordinator+Overlord, 1× Broker, 1× Historical, 1× Router, 1× MiddleManager).
#### Configurations in use
Historical `runtime.properties` (intentionally constrained to force saturation; production-realistic values would be larger but the same code path triggers under sufficient load):
```properties
druid.server.http.numThreads=4
druid.server.http.enableRequestLimit=true
druid.server.http.maxIdleTime=PT5S
druid.processing.numThreads=1
druid.processing.numMergeBuffers=2
druid.processing.buffer.sizeBytes=64MiB
```
Broker `runtime.properties`:
```properties
druid.broker.http.numConnections=20
druid.broker.http.readTimeout=PT2M
druid.broker.http.unusedConnectionTimeout=PT4S
druid.server.http.numThreads=20
```
Debug logging on Broker / Historical:
- `org.apache.druid.jetty.RequestLog=DEBUG`
- `org.apache.druid.client.DirectDruidClient=DEBUG`
- `org.apache.druid.client.JsonParserIterator=DEBUG`
- `org.eclipse.jetty.server.handler.ErrorHandler=DEBUG`
- `org.eclipse.jetty.util.thread.QueuedThreadPool=DEBUG`
#### Steps to reproduce
1. Apply the constrained Historical config above and restart the Historical.
2. Drive a concurrent load against the Router using a small set of queries on the `wikipedia` datasource (e.g., 1000 requests at concurrency 200).
3. Observe HTTP 500 responses on the client containing `JsonParseException`, and matching HTTP 429 in the Historical request log.
A direct probe against the Historical (or any inspection of `historical.log`) shows the HTML body:
```
HTTP/1.1 429 Too Many Requests
Content-Type: text/html;charset=ISO-8859-1
Error 429 Too Many Requests
HTTP ERROR 429 Too Many Requests
...
```
#### Error message / stack trace
Broker:
```
WARN [ForkJoinPool-1-worker-3] org.apache.druid.client.JsonParserIterator -
Query [5710bfb8-ade6-4d13-8211-641c7e2f512f] to host [localhost:8083] interrupted
com.fasterxml.jackson.core.JsonParseException: Invalid type marker byte 0x3c
for expected value token
at [Source: (SequenceInputStream); byte offset: #1]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:724)
...
at org.apache.druid.client.JsonParserIterator.init(JsonParserIterator.java:216)
at org.apache.druid.client.DirectDruidClient.run(DirectDruidClient.java:381)
```
Client (through Router):
```json
HTTP/1.1 500 Internal Server Error
{"error":"Unknown exception",
"errorClass":"com.fasterxml.jackson.core.JsonParseException",
"host":"localhost:8083",
"errorCode":"legacyQueryException",
"persona":"OPERATOR",
"category":"RUNTIME_FAILURE"}
```
#### Debugging done
Cross-checked against Druid sources:
- `server/src/main/java/org/apache/druid/server/initialization/jetty/LimitRequestsFilter.java`
emits `((HttpServletResponse) response).sendError(429, "Too Many Requests")`,
which dispatches to Jetty's default `ErrorHandler` — producing HTML.
- `server/src/main/java/org/apache/druid/server/initialization/jetty/JettyServerModule.java` (around line 469)
only customizes the `ErrorHandler` when `druid.server.http.showDetailedJettyErrors=false`,
and even then it merely clears `RequestDispatcher.ERROR_EXCEPTION` and delegates to
`super.handle(...)`. The body rendering remains Jetty's default (HTML when no
`Accept: application/json` header is sent, which is the case for `DirectDruidClient`).
- `server/src/main/java/org/apache/druid/client/DirectDruidClient.java` does not check
`Content-Type`. The `InputStream` is handed straight to
`JsonParserIterator.init()` (`server/src/main/java/org/apache/druid/client/JsonParserIterator.java`),
which calls `objectMapper.getFactory().createParser(is); jp.nextToken();` —
exactly where Jackson trips over the leading `<`.
- The `QoSFilter` registered in
`services/src/main/java/org/apache/druid/cli/QueryJettyServerInitializer.java`
produces HTTP 503 on suspend timeout via the same code path, so 503 responses can
trigger the identical failure.
- `docs/configuration/index.md` description of `druid.server.http.showDetailedJettyErrors`
("the JSON response only includes...") is misleading: the default error handler still
performs `Accept`-header based content negotiation and falls back to HTML for clients
that do not advertise `application/json`.
Contributor guide
Research direction
Reproduce the failure with the constrained Historical settings and concurrent Router load, then read DirectDruidClient.java and JsonParserIterator.java alongside LimitRequestsFilter.java, JettyServerModule.java, and QueryJettyServerInitializer.java. Verify how HTML 429/503 responses are handled, and define done as preventing the Jackson parse failure while returning a meaningful capacity or rate-limit error to the client.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100