Remote read adding external labels might lead to unsorted response

What did you do?

/api/v1/read is supposed to always give a sorted series set as a response. Consider the following:

External labels are region: foo.

Series without any external labels:

aaa{instance="bbb", job="etcd-prometheus-exporter", service="etcd-prometheus-exporter", toposphere="test1"} 1
aaa{instance="bbb", job="etcd-prometheus-exporter", region="bar", service="etcd-prometheus-exporter", toposphere="test1"} 0
aaa{instance="bbb", job="etcd-prometheus-exporter", region="bar", service="etcd-prometheus-exporter", toposphere="test2"} 0
aaa{instance="bbb", job="etcd-prometheus-exporter", region="foo", service="etcd-prometheus-exporter", toposphere="test1"} 1
aaa{instance="bbb", job="etcd-prometheus-exporter", region="foo", service="etcd-prometheus-exporter", toposphere="test2"} 1

https://github.com/prometheus/prometheus/blob/03e549cc39f017f130028737fa71f8abece6d736/storage/remote/codec.go#L230 here is where Prometheus adds region: foo if it does not exist while iterating.

What did you expect to see?

I expected to see an error? that the response cannot contain the same label set. Not sure, though.

What did you see instead? Under which circumstances?

The result is the following:

aaa{instance="bbb", job="etcd-prometheus-exporter", region="foo", service="etcd-prometheus-exporter", toposphere="test1"} 1
aaa{instance="bbb", job="etcd-prometheus-exporter", region="bar", service="etcd-prometheus-exporter", toposphere="test1"} 0
aaa{instance="bbb", job="etcd-prometheus-exporter", region="bar", service="etcd-prometheus-exporter", toposphere="test2"} 0
aaa{instance="bbb", job="etcd-prometheus-exporter", region="foo", service="etcd-prometheus-exporter", toposphere="test1"} 1
aaa{instance="bbb", job="etcd-prometheus-exporter", region="foo", service="etcd-prometheus-exporter", toposphere="test2"} 1

System information

No response

Prometheus version

No response

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response

Start at storage/remote/codec.go around line 230 and reproduce the reported case through the /api/v1/read endpoint. Determine the intended behavior when external labels create duplicate or unsorted label sets; the work is done when that behavior is explicit and the response handling is covered by an appropriate regression test.