badGateway errors when using AsyncHTTPClient version > 1.6.4 with Vapor 4 project

AsyncHTTPClient commit hash:

ec2e080d7011a81bd67f10bf41efe6104d7799d6

Swift Version:

swift-driver version: 1.26.21 Apple Swift version 5.5.2 (swiftlang-1300.0.47.5 clang-1300.0.29.30)
Target: x86_64-apple-macosx12.0

Context

We have a Vapor 4 project that has been running in production close to a year now. This project communicates with 3rd party APIs using a Client which has worked fine so far. Vapors Client (EventLoopHTTPClient to be specific) wraps a HTTPClient from AsyncHTTPClient.

Recently I was tasked with implementing some new features and as part of this I updated all dependencies so I could take advantage of async/await.

Issue

After updating Vapor and other dependencies I noticed that the existing calls to 3rd party APIs stopped working. Calling the endpoints would return a .badGateway error instead. I could write this off as errors in the 3rd party API if it was one of them, but when all of them started to fail...that indicated that the error was on my side 😅

Workaround

I managed to get the endpoints working again by manually adding an .exact dependency to async-http-client in my Package.swift file like so:

.package(url: "https://github.com/swift-server/async-http-client.git", .exact("1.6.4"))

If I upgrade to anything above 1.6.4, I start seeing the .badGateway errors on my existing Client calls (which are using EventLoopFutures btw.)

I can see that AsyncHTTPClient 1.7.0 introduces automatic HTTP/2 support by default and I wonder if that has broken something in my setup.

Additional Workaround

As mentioned in this issue, setting the httpVersion manually also does the trick and means that you do not have to stay on 1.6.4.

So, somewhere in your config before you start using the client add this:
app.http.client.configuration.httpVersion = .http1Only

Additional Observations

• the jobs I added are using async/await and that seems to work (they are speaking to a different 3rd party though so that may be the reason why)
• one of the existing jobs does an initial login to obtain an access token which is then used for the actual call immediately after. The login call works, but the following call to get relevant content fails
• I can call all of the "failing" endpoints using cURL from the same machine and get a successful response back

I know that this is probably a case of me doing something wrong somehow but I hope you can assist me in pinpointing where the issue lies 😄

Thank you for your time.

Start with the AsyncHTTPClient 1.7.0 release notes and the existing EventLoopHTTPClient calls described in the issue. Reproduce the badGateway responses with dependencies above 1.6.4, then compare them with Package.swift pinned to 1.6.4 and with app.http.client.configuration.httpVersion set to .http1Only. Done means the affected calls work without requiring the version pin or manual HTTP/1 configuration.