Instance stays when priority is changed

When I set the snapshot with the instance in locality dc1 with priority 0 and then set another one with instance in locality dc1 with priority 1, but without the first one, there are two instances in Envoy.

I've written following test
https://github.com/jakubdyszkiewicz/java-control-plane/commit/39b76bc71fe48ab8f727d66714ef3f3f427dc039

But then it turned out that the problem seems to be even more general.
When I set a snapshot with endpoint with priority 0, then set another one with priority 1, but without the first one - both stays in Envoy.
https://github.com/jakubdyszkiewicz/java-control-plane/blob/a2a6b4b5498fa1de734ed76aafcdf768067f525a/server/src/test/java/io/envoyproxy/controlplane/server/MoveInstanceToAnotherLocalityTest.java

Not sure whether this is an issue with java-control-plane or Envoy itself or this is just an expected behaviour.

I've seen issues in Envoy that may be connected with this
https://github.com/envoyproxy/envoy/issues/3327
https://github.com/envoyproxy/envoy/issues/3312

@snowp it would be helpful if you could take a look at this, because you fixed above issues. Thank you.