CurlAsyncHTTPClient doesn't seem to handle the requests in parallel

Hi,
I'm on tornado 6.1, python 3.9 running on an alpine container 4 CPU with 4 GB of Ram, and I'm seeing a weird behavior when calling a remote service using
response = await AsyncHTTPClient().fetch(http_request)

What I'm seeing is that when I flooding my service with requests (~ 250 requests in about 2-3 seconds) that have to call this remote service, the requests seem to be queued, waiting a lot of time for getting the connection or waiting for the response from the remote server, although the remote server is not loaded at all, and should be able to handle all these requests at once.

Is there anything limiting me in processing all these requests in parallel, at the libcurl / pycurl libs levels? Is there any way to force CurlAsyncHTTPClient to execute these request in parallel ?

AsyncHTTPClient is initialized like this:

max_clients = 1000  # increasing this value doesn't seem to have any impact in my system performance
http_client_class = 'tornado.curl_httpclient.CurlAsyncHTTPClient'
AsyncHTTPClient.configure(http_client_class, max_clients=max_clients)

These are the details from some of the longest responses from the remote server:

http://dialog-tool-app-server-envoy:80/dialog-tool/v1/projects/512020/variable-expressions/5e1718fa-8d51-490f-910f-58345517b02c 44155.79ms {'queue': 1.430511474609375e-05, 'namelookup': 0.006512, 'connect': 10.055009, 'appconnect': 0.0, 'pretransfer': 10.055321, 'starttransfer': 42.601884, 'total': 42.601966, 'redirect': 0.0}

http://dialog-tool-app-server-envoy:80/dialog-tool/v1/projects/512020/variable-expressions/9b68dc03-ee31-41c8-b45e-be11b9c41e0a 25419.50ms {'queue': 1.239776611328125e-05, 'namelookup': 0.000131, 'connect': 0.000131, 'appconnect': 0.0, 'pretransfer': 0.000467, 'starttransfer': 10.867779, 'total': 10.867838, 'redirect': 0.0}

The very first processed requests take about 1.5 seconds to complete:

http://dialog-tool-app-server-envoy:80/dialog-tool/v1/projects/512020/variable-expressions/3ee1ecbd-6d73-4ca5-828a-8a8b8e1ce5e9 1543.81ms {'queue': 1.1444091796875e-05, 'namelookup': 5.6e-05, 'connect': 5.6e-05, 'appconnect': 0.0, 'pretransfer': 0.000432, 'starttransfer': 1.542342, 'total': 1.542444, 'redirect': 0.0}",,,,

Start with the AsyncHTTPClient.configure call and the CurlAsyncHTTPClient behavior described in the issue, then inspect how the configured max_clients value relates to libcurl and pycurl concurrency. Reproduce the reported load pattern and compare queue, connect, starttransfer, and total timings; done means identifying and documenting the limiting factor or confirming a reproducible client defect.