apache / apache/trafficserver

9.2.x Unexpected connection errors when proxying larger request bodies

Open
#10,393 2 comments 0 reactions 1 assignee Assigned to @shinrich View on GitHub
Bug HTTP Stale Support
Dominant language
C++
Stars
2k
Forks
874
Avg merge
6d 15h
Merged PRs (30d)
46

Description

When upgrading from 9.1.4 to 9.2.x, I've observed some issues with requests failing in unexpected ways that they didn't previously in 9.1.4. I'm not really certain what's happening, so it's a bit difficult to give a summary, but it seems like perhaps Traffic Server 9.2.x is closing connections to clients in front of Traffic Server unexpectedly too early in certain cases. My only theory is that it's somehow related to larger request bodies (and maybe specific to PUT requests), and maybe only when these larger request bodies are still being streamed after the origin server generates an (expected) error? But again, not really sure.

Here's a more detailed example of how this is pretty reproducible in all versions of 9.2.0-9.2.2, and also demonstrates how this didn't happen in 9.1.4. The basic reproducible case I've narrowed this down to looks like this:

```
[nginx proxy] => [trafficserver] => [nginx server]
```

1. The `nginx proxy` layer does *not* have a maximum request body size.
2. The underlying `nginx server` component *is* setup with a maximum request body size. If a client sends a request body that exceeds this size, then nginx returns a `413 Request Entity Too Large` error.

The basic issue I'm seeing is that if a client exceeds this request body size at the `nginx server` origin layer then Traffic Server 9.2+ seems to behave in unexpected ways:

1. **In Traffic Server 9.1.4:** The `nginx proxy` layer (and client making the request) reliably receives the `413 Request Entity Too Large` error that the origin `nginx server` layer generates and is proxied via TrafficServer.
2. **In Traffic Server 9.2.2:** The `nginx proxy` layer maybe 50% of the time receives the expected 413 error (from the `nginx server` origin), but then the other 50% of the time the `nginx proxy` ends up reporting a `502 Bad Gateway` error which is generated by nginx due to an apparent communication errors with `trafficserver`. This seems to indicate that the connection from `nginx proxy` to `trafficserver` is being closed unexpectedly by Traffic Server too early before the `413` error can be proxied back successfully.

Here is a repo that contains a minimal reproduction of this along more detailed steps: https://github.com/GUI/trafficserver-debugging This issue appears to be present using all default Traffic Server configuration, so there's no custom Traffic Server configuration other than proxying to the underlying server. See the repo's README for exact steps to reproduce an more examples of the expected output in Traffic Server 9.1.4 versus the new behavior that's more erratic in Traffic Server 9.2.2.

The short version is that Traffic Server 9.1.4 will always return the expected `413 Request Entity Too Large` that is proxied from the underlying origin server, but when Traffic Server 9.2.x is in the middle, then it will randomly lead to nginx's connections to Traffic Server failing and the `nginx proxy` layer generates `502 Bad Gateway` errors.

A few notes I've observed:

- It happens more readily if the request body size is bigger (eg, more than a couple MBs).
- Strangely, I can reproduce it reliably for PUT requests with a body, but not POST requests.
- In tcpdumps, I've observed TCP RSTs under TrafficServer 9.2.x during these situations where there don't appear to be any RSTs in 9.1.x.
- I've been able to reproduce this in both 9.2.0 and 9.2.2, so it seems like it's related to some change between 9.1.4 to 9.2.0.

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.