Incorrect handling of ESI calls initiated after POST request
- Dominant language
- No language data
- Stars
- 126
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
# Bug description
knot.x version: `1.4.0`
Email signup is one of the features on my website. User initiates `POST` request which bypasses knot.x completely. The response contains ESI markup, so CDN triggers sub-requests, which go via knot.x instance. knot.x returns 500 for each subrequest call.
This situation doesn't happen when the entire flow starts with `GET` request.
Here's a diagram with end-to-end HTTP flow:
# Steps to reproduce
1. Send a `POST` request with `Content-Type: application/x-www-form-urlencoded` header
2. Make sure response contains ESI markup
3. ESI sub-requests should go via knot.x (they inherit request headers from original request - only those that came from the user, `Content-Type` is one of them)
4. Each ESI call ends with 500 response
Response headers:
```
HTTP/1.1 500 java.nio.file.AccessDeniedException: /file-uploads
Content-Type: text/html
Connection: close
Content-Length: 9372
```
Response body:
```
java.nio.file.AccessDeniedException: /file-uploads
body {
margin: 0;
padding: 80px 100px;
font: 13px "Helvetica Neue", "Lucida Grande", "Arial";
background: #ECE9E9 -webkit-gradient(linear, 0% 0%, 0% 100%, from(#fff), to(#ECE9E9));
background: #ECE9E9 -moz-linear-gradient(top, #fff, #ECE9E9);
background-repeat: no-repeat;
color: #555;
-webkit-font-smoothing: antialiased;
}
h1, h2, h3 {
margin: 0;
font-size: 22px;
color: #343434;
}
h1 em, h2 em {
padding: 0 5px;
font-weight: normal;
}
h1 {
font-size: 60px;
}
h2 {
margin-top: 10px;
}
h3 {
margin: 5px 0 10px 0;
padding-bottom: 5px;
border-bottom: 1px solid #eee;
font-size: 18px;
}
ul {
margin: 0;
padding: 0;
}
ul li {
margin: 5px 0;
padding: 3px 8px;
list-style: none;
}
ul li:hover {
cursor: pointer;
color: #2e2e2e;
}
p {
line-height: 1.5;
}
a {
color: #555;
text-decoration: none;
}
a:hover {
color: #303030;
}
#stacktrace {
margin-top: 15px;
}
.directory h1 {
margin-bottom: 15px;
font-size: 18px;
}
Matron!
500 java.nio.file.AccessDeniedException: /file-uploads
- io.vertx.core.file.impl.FileSystemImpl$11.perform(FileSystemImpl.java:678)
- io.vertx.core.file.impl.FileSystemImpl$11.perform(FileSystemImpl.java:660)
- io.vertx.core.file.impl.FileSystemImpl.mkdirsBlocking(FileSystemImpl.java:248)
- io.vertx.ext.web.handler.impl.BodyHandlerImpl$BHandler.makeUploadDir(BodyHandlerImpl.java:171)
- io.vertx.ext.web.handler.impl.BodyHandlerImpl$BHandler.(BodyHandlerImpl.java:138)
- io.vertx.ext.web.handler.impl.BodyHandlerImpl.handle(BodyHandlerImpl.java:72)
- io.vertx.ext.web.handler.impl.BodyHandlerImpl.handle(BodyHandlerImpl.java:42)
- io.vertx.reactivex.ext.web.handler.BodyHandler.handle(BodyHandler.java:74)
- io.vertx.reactivex.ext.web.handler.BodyHandler.handle(BodyHandler.java:37)
- io.vertx.reactivex.ext.web.Route$1.handle(Route.java:155)
- io.vertx.reactivex.ext.web.Route$1.handle(Route.java:153)
- io.vertx.ext.web.impl.RouteImpl.handleContext(RouteImpl.java:219)
- io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:120)
- io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:133)
- io.vertx.ext.web.handler.impl.CookieHandlerImpl.handle(CookieHandlerImpl.java:66)
- io.vertx.ext.web.handler.impl.CookieHandlerImpl.handle(CookieHandlerImpl.java:42)
- io.vertx.reactivex.ext.web.handler.CookieHandler.handle(CookieHandler.java:73)
- io.vertx.reactivex.ext.web.handler.CookieHandler.handle(CookieHandler.java:36)
- io.vertx.reactivex.ext.web.Route$1.handle(Route.java:155)
- io.vertx.reactivex.ext.web.Route$1.handle(Route.java:153)
- io.vertx.ext.web.impl.RouteImpl.handleContext(RouteImpl.java:219)
- io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:120)
- io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:133)
- io.vertx.reactivex.ext.web.RoutingContext.next(RoutingContext.java:128)
- io.knotx.server.SupportedMethodsAndPathsHandler.handle(SupportedMethodsAndPathsHandler.java:52)
- io.knotx.server.SupportedMethodsAndPathsHandler.handle(SupportedMethodsAndPathsHandler.java:27)
- io.vertx.reactivex.ext.web.Route$1.handle(Route.java:155)
- io.vertx.reactivex.ext.web.Route$1.handle(Route.java:153)
- io.vertx.ext.web.impl.RouteImpl.handleContext(RouteImpl.java:219)
- io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:120)
- io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:133)
- io.vertx.reactivex.ext.web.RoutingContext.next(RoutingContext.java:128)
- io.knotx.server.KnotxHeaderHandler.handle(KnotxHeaderHandler.java:41)
- io.knotx.server.KnotxHeaderHandler.handle(KnotxHeaderHandler.java:23)
- io.vertx.reactivex.ext.web.Route$1.handle(Route.java:155)
- io.vertx.reactivex.ext.web.Route$1.handle(Route.java:153)
- io.vertx.ext.web.impl.RouteImpl.handleContext(RouteImpl.java:219)
- io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:120)
- io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:133)
- io.vertx.ext.web.handler.impl.LoggerHandlerImpl.handle(LoggerHandlerImpl.java:178)
- io.vertx.ext.web.handler.impl.LoggerHandlerImpl.handle(LoggerHandlerImpl.java:47)
- io.vertx.reactivex.ext.web.handler.LoggerHandler.handle(LoggerHandler.java:73)
- io.vertx.reactivex.ext.web.handler.LoggerHandler.handle(LoggerHandler.java:36)
- io.vertx.reactivex.ext.web.Route$1.handle(Route.java:155)
- io.vertx.reactivex.ext.web.Route$1.handle(Route.java:153)
- io.vertx.ext.web.impl.RouteImpl.handleContext(RouteImpl.java:219)
- io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:120)
- io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:133)
- io.vertx.ext.web.impl.RouterImpl.accept(RouterImpl.java:79)
- io.vertx.reactivex.ext.web.Router.accept(Router.java:94)
- io.knotx.server.KnotxServerVerticle.routeSafe(KnotxServerVerticle.java:181)
- io.knotx.server.KnotxServerVerticle.lambda$start$8(KnotxServerVerticle.java:164)
- io.vertx.reactivex.core.http.HttpServer$1.handle(HttpServer.java:111)
- io.vertx.reactivex.core.http.HttpServer$1.handle(HttpServer.java:109)
- io.vertx.core.http.impl.Http1xServerConnection.processMessage(Http1xServerConnection.java:453)
- io.vertx.core.http.impl.Http1xServerConnection.handleMessage(Http1xServerConnection.java:144)
- io.vertx.core.http.impl.HttpServerImpl$ServerHandlerWithWebSockets.handleMessage(HttpServerImpl.java:666)
- io.vertx.core.http.impl.HttpServerImpl$ServerHandlerWithWebSockets.handleMessage(HttpServerImpl.java:619)
- io.vertx.core.net.impl.VertxHandler.lambda$channelRead$1(VertxHandler.java:146)
- io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:337)
- io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:195)
- io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:144)
- io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
- io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
- io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
- io.vertx.core.http.impl.HttpServerImpl$Http2UpgradeHandler.channelRead(HttpServerImpl.java:968)
- io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
- io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
- io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
- io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
- io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
- io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
- io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
- io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
- io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
- io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
- io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
- io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
- io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
- io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
- io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
- io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
- io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
- io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
- io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
- java.lang.Thread.run(Thread.java:748)
```
# Expected behavior
ESI calls are regular `GET` requests. The only difference in comparison to other situations is the fact that sub-requests inherit parent request headers, so CDN sends `GET` with `Content-Type: application/x-www-form-urlencoded`.
knot.x should either reject such request with 400 status code or ignore `Content-Type` header sent with `GET` request, as it doesn't make any sense.
# Screenshots
N/A
# Additional context
The following `curl` command was enough to reproduce the problem:
```
$ curl localhost:8092/path/to/esi/snippet.html -H "Content-Type: application/x-www-form-urlencoded"
```
Those 500s are visible in `knotx-access.log` file, but `knotx.log` stays empty despite of the fact I increased log level to `DEBUG`.
```
10.251.35.221 - - [Mon, 14 Jan 2019 22:59:25 GMT] "GET /path/to/esi/snippet.html HTTP/1.1" 500 8931 "http://example.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
```
It could be related to #321
Possible workaround - "sanitize" HTTP request before it gets processed by knot.x and remove `Content-Type` from `GET` requests.
Contributor guide
Research direction
Reproduce the failure with the provided curl command and inspect the request path through Vert.x BodyHandlerImpl, SupportedMethodsAndPathsHandler, and KnotxHeaderHandler as shown in the stack trace. Compare handling of a GET with and without Content-Type: application/x-www-form-urlencoded, then verify that the chosen behavior avoids the 500 response and is covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100