[oCIS] Scaling oCIS in kubernetes causes requests to fail Timebox 8PD

During loadtests we seem to be losing requests. We have identified several possible causes:
## 1. when a new pod is added it does not seem to receive traffic
This might be caused by clients not picking up the new service. One reason would be that the same grpc connection is reused. We need to make sure that every service uses the a selector.Next() call to get a fresh client from the registry.

## 2. when a pod is shut down because kubernetes moves it to a different node or it is descheduled it still receives traffic
This might be caused by latency. The client got a grpc client with selector.Next() but then the pod was killed before the request reached it. We should retry requests, but the grpc built in retry mechanism would need to know all possible services. That is not how the reva pool works.

We could configure the grpc connection to retry requests:
```go

var retryPolicy = `{
"methodConfig": [{
// config per method or all methods under service
"name": [{"service": "grpc.examples.echo.Echo"}],
"waitForReady": true,

"retryPolicy": {
"MaxAttempts": 4,
"InitialBackoff": ".01s",
"MaxBackoff": ".01s",
"BackoffMultiplier": 1.0,
// this value is grpc code
"RetryableStatusCodes": [ "UNAVAILABLE" ]
}
}]
}`

conn, err := grpc.Dial(
address,
grpc.WithTransportCredentials(cred),
grpc.WithDefaultServiceConfig(retryPolicy),
grpc.WithDefaultCallOptions(
grpc.MaxCallRecvMsgSize(maxRcvMsgSize),
),
grpc.WithStatsHandler(otelgrpc.NewClientHandler(
otelgrpc.WithTracerProvider(
options.tracerProvider,
),
otelgrpc.WithPropagators(
rtrace.Propagator,
),
)),
)
```
but they would just try the same ip. To actually send requests to different servers, aka [client side load balancing](https://github.com/grpc/grpc-go/blob/master/examples/features/load_balancing/README.md) we would have to add sth. like:
```go
// Make another ClientConn with round_robin policy.
roundrobinConn, err := grpc.Dial(
fmt.Sprintf("%s:///%s", exampleScheme, exampleServiceName),
grpc.WithDefaultServiceConfig(`{"loadBalancingConfig": [{"round_robin":{}}]}`), // This sets the initial balancing policy.
grpc.WithTransportCredentials(insecure.NewCredentials()),
)
```

The load balancing works based on [name resolving](https://github.com/grpc/grpc-go/blob/master/examples/features/name_resolving/README.md).

We could add all this to the reva pool ... or we use a go micro grpc client that already implements a pool, integrates with the service registry and can do retry, backoff and whatnot. But this requires generating micro glients for the cs3 api using `github.com/go-micro/generator/cmd/protoc-gen-micro`

## 3. pod readyness and health endpoints do not reflect the actual state of the pod
Currently, the `/healthz` and `/readyz` endpoints are independent from the actual service implementation. But some services need some time to be ready or flush all requests on shutdown. This also needs to be investigated.
For ready we could use a channel to communicate between the actual handler and the debug handler.
And AFAIR @rhafer mentioned we need to take care of shutdown functions ... everywhere.

## 4. the services are needlessly split into separate pods
Instead of startinf a pod for every service we should aggregate all processes that are involved in translating a request until they reach a storage provider:
* proxy should stay alone as it is the first service that is hit by traffic. and we may need it to shard the userbase of large instances by routing requests to a specific shard
* frontend, webdav, ocs & graph -> gateway & auth providers are all stateless and should run in a single frontend pod
* storage-system might go together with user and group providers
* strorage-users does the bulk of the work this makes sense to put into a pod (actually this already combines a storageprovider and a dataprovider which we should maybe even split? one is for metadata, the other for blob transfer)
* sharing ... might even go into the frontend

The services should use localhost or even unix sockets to talk to each other. go can very efficiently use the resources in a pod an handle requests concurrently. We really only create a ton of overhead that stresses the kubernetes APIs and can be reduced.