Pinot query timeout due to the broker waiting for a single non-responsive server

Does Pinot broker need to wait the responses of all the servers before it can return the results? We found a single hanging server (probably due to GC) make queries to return after timeout. Restarting the server fixed the issue. A single server failure should not cause timeout when we have multiple copies of the data. Is there any step in query execution requires all servers to respond?