Streaming Result for Query/QueryRange HTTP API

In Thanos we are moving forward with query sharding, parallelization and pushdown ideas 💪🏽 Recently, we pushed a change that is capable of safely pushdown certain aggregations to Prometheus (https://github.com/thanos-io/thanos/pull/4917).

What we do in sidecar now, for certain "safe" functions like min, max, group, min_over_time, and max_over_time instead of fetching all series through remote read, we change to HTTP Query API. We build PromQL from select hints we get from root Querier and perform query. Then we transform the Query result to Thanos GRPC StoreAPI.

Now, we have efficiency problem with Query API, because it's not streamed (something I was moaning about for long time 🙈). This means we unnecessary wait for full response payload, instead of streaming each resulted series by series like we do with remote read. AFAIK for most cases PromQL performs series by series calculations, so first series is calculated before next one, so that fits server part of this API too.

Now, to solve our use case, there are two options we could do to improve Prometheus for these use cases and actually win in other fields too.

A) Introduce streaming (per series) for query and query range Prometheus HTTP APIs. We could do this exactly as we did with remote read. Don't stream by default and opt-in using header negotiation for streamed JSON using chunked JSON logic we use with profobuf in new remote read. It's not straightforward as our current JSON result with this data format was never attempted to be streamed.
B) Allow remote-read to perform PromQL using hints. It's not hard to implement, but quite weird as there is strong overlap now between Query + Query range and remote write that can do PromQL. Plus it's easy right now to make mistake and pass wrong hints (aggregations/function that are NOT safe to be pushed down).

I would vote for A. I believe streaming response on query would help with many other use cases (e.g Prometheus -> Grafana interaction and Querier -> Query frontend in Thanos).

If we are ok to pursue A I could try to work with community on proposing exact format of JSON streamed API.

cc @tomwilkie @roidelapluie @juliusv @fpetkovski @brian-brazil @moadz @RichiH

Start by reviewing the Query and QueryRange HTTP API behavior described in the issue, along with the existing remote-read streamed JSON and chunked protobuf approach. The issue does not name implementation files or tests; work would first require agreeing on the streamed JSON format and opt-in header negotiation with the community. Done would mean an agreed design and an implemented streaming response for the proposed use case.