Consistent hashing for proxy load balancers

I'm trying to create a Buildkit cluster in k8s, with ingress-nginx as a gRPC proxy load balancer. I got it working when the k8s service is backed by a single pod, but as soon as I scale-up, I start getting errors like below. The more instances, the greater the odds of getting an error.

• error: failed to solve: DeadlineExceeded: no active session for die6thd3zowmxf946m911t820: context deadline exceeded
• error: failed to receive status: rpc error: code = Unknown desc = no such job v0p7pbqubqagkgydbxqes231q

I tried to use Nginx's consistent hashing LB algorithm for my Buildkit upstream servers, but that requires something common across related requests to use as the hashing key.

I gave the X-Docker-Expose-Session-Uuid header a try, but that doesn't seem to be included in all requests. I can see that it is included in the /moby.buildkit.v1.Control/Session requests, but not in /moby.buildkit.v1.Control/Status or /moby.buildkit.v1.Control/Solve. I don't know much about gRPC, but I guess that is by design.

Here is what I see in my Nginx logs:

1. /moby.buildkit.v1.Control/Status hits pod_1
2. /moby.buildkit.v1.Control/Solve hits pod_1
3. /moby.buildkit.v1.Control/Session hits pod_2
4. /opentelemetry.proto.collector.trace.v1.TraceService/Export hits pod_1

After those requests, buildctl crashes with error: failed to solve: DeadlineExceeded: no active session for os0bz8rfotds633blqc6fko8a: context deadline exceeded.

So, I ask: is there any request information (that wouldn't require unpacking the gRPC "body") that I could reliably use for consistent hash? If not, would it be possible to introduce something like that to allow for this sort of load balancing? A new header, perhaps?

Before someone comes with the suggestion of using source IP or something like X-Forwarded-For, I thought about that but it wouldn't help me much... I don't want requests from the same client to always hit the same pod, because:

1. the load generated by my multiple clients is far from even
2. I'm not worried about daemon local caching, since I'm pushing my cache to a remote registry and am comfortable with the performance penalties.

The issue names no source files, tests, or entry points. Start by tracing the gRPC Control methods and how the X-Docker-Expose-Session-Uuid header is handled across Status, Solve, and Session requests. Done means establishing a reliable load-balancing key or defining a supported API change, with coverage for the related request flow.