Request unauthorized for SQL search-attributes (claims are not passed)

Expected Behavior

I can execute CRUD commands with admin and non admin x-search-attributes without errors.

Actual Behavior

I have devl:admin and temporal-system:admin roles assigned. I use only sql, postgres12 driver for "default" and "visibility" dbs.

When I execute one of x-search-attributes command inside admintools pod it fails with error:

I use devl namespace as an example of not working scenario

tctl --namespace=devl admin cluster get-search-attributes
Error: Unable to get search attributes.
Error Details: rpc error: code = PermissionDenied desc = Request unauthorized.

When I run this command with not existing namespace I get different error:

Error Details: rpc error: code = NotFound desc = Namespace searchtest2 is not found.

Cluster is working:

tctl cluster health
temporal.api.workflowservice.v1.WorkflowService: SERVING

I tested other admin commands (provided auth token as an env variable) related to cluster info (tctl --namespace=devl admin cluster describe, tctl --namespace=devl admin cluster list ), namespaces (tctl --namespace=devl namespace list, tctl --namespace=devl namespace describe), etc. and they are working. Only search-attributes has this problem. Non-admin command (tctl --namespace=devl cluster get-search-attributes) works as expected. Unfortunately only admin command allows to add attributes.
Update: The newest Temporal version allows to add new search attributes, admin commands are marked as deprecated.

Log from frontend pod:

 {"level":"error","ts":"2024-10-15T13:59:29.365Z","msg":"service failures","operation":"AdminGetSearchAttributes","wf-namespace":"devl","error":"Unable to get namespace devl info with error: Request unauthorized.","logging-call-at":"telemetry.go:411","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).handleError\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/telemetry.go:411\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).UnaryIntercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/telemetry.go:202\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.temporal.io/server/service/frontend.(*RedirectionInterceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/frontend/redirection_interceptor.go:187\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.temporal.io/server/common/authorization.(*Interceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/authorization/interceptor.go:181\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.temporal.io/server/service/frontend.GrpcServerOptionsProvider.NewServerMetricsContextInjectorInterceptor.func2\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.51.0/interceptor.go:315\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/namespace_logger.go:85\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).NamespaceValidateIntercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/namespace_validator.go:135\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.temporal.io/server/common/utf8validator.(*Validator).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/utf8validator/validate.go:182\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.temporal.io/server/service/frontend.GrpcServerOptionsProvider.NewServiceErrorInterceptor.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/grpc.go:178\ngoogle.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1177\ngo.temporal.io/server/api/adminservice/v1._AdminService_GetSearchAttributes_Handler\n\t/home/runner/work/docker-builds/docker-builds/temporal/api/adminservice/v1/service_grpc.pb.go:1062\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1369\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1780\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1019"}

Stacktrace from command:

Error: Unable to get search attributes.
Error Details: rpc error: code = Unavailable desc = Unable to get namespace devl info with error: Request unauthorized.
Stack trace:
goroutine 1 [running]:
runtime/debug.Stack()
	/opt/hostedtoolcache/go/1.21.11/x64/src/runtime/debug/stack.go:24 +0x5e
runtime/debug.PrintStack()
	/opt/hostedtoolcache/go/1.21.11/x64/src/runtime/debug/stack.go:16 +0x13
github.com/temporalio/tctl/cli_curr.printError({0x1990e48, 0x20}, {0x1dd3940, 0xc00051c000})
	/home/runner/work/docker-builds/docker-builds/tctl/cli_curr/util.go:393 +0x218
github.com/temporalio/tctl/cli_curr.ErrorAndExit({0x1990e48?, 0x1dfaf50?}, {0x1dd3940?, 0xc00051c000?})
	/home/runner/work/docker-builds/docker-builds/tctl/cli_curr/util.go:404 +0x25
github.com/temporalio/tctl/cli_curr.AdminGetSearchAttributes(0xc0005371e0)
	/home/runner/work/docker-builds/docker-builds/tctl/cli_curr/admin_cluster_search_attributes_commands.go:157 +0x89
github.com/temporalio/tctl/cli_curr.newAdminClusterCommands.func3(0xc0005371e0?)
	/home/runner/work/docker-builds/docker-builds/tctl/cli_curr/admin.go:496 +0x13
github.com/urfave/cli.HandleAction({0x16a4dc0?, 0x1a3db98?}, 0x15?)
	/home/runner/go/pkg/mod/github.com/urfave/cli@v1.22.10/app.go:526 +0x75
github.com/urfave/cli.Command.Run({{0x1978c96, 0x15}, {0x0, 0x0}, {0xc0006cb080, 0x1, 0x1}, {0x198e717, 0x1f}, {0x0, ...}, ...}, ...)
	/home/runner/go/pkg/mod/github.com/urfave/cli@v1.22.10/command.go:173 +0x63e
github.com/urfave/cli.(*App).RunAsSubcommand(0xc000239880, 0xc000536f20)
	/home/runner/go/pkg/mod/github.com/urfave/cli@v1.22.10/app.go:405 +0xe07
github.com/urfave/cli.Command.startApp({{0x195d3e0, 0x7}, {0x0, 0x0}, {0xc0006cb220, 0x1, 0x1}, {0x198c21d, 0x1e}, {0x0, ...}, ...}, ...)
	/home/runner/go/pkg/mod/github.com/urfave/cli@v1.22.10/command.go:378 +0xb58
github.com/urfave/cli.Command.Run({{0x195d3e0, 0x7}, {0x0, 0x0}, {0xc0006cb220, 0x1, 0x1}, {0x198c21d, 0x1e}, {0x0, ...}, ...}, ...)
	/home/runner/go/pkg/mod/github.com/urfave/cli@v1.22.10/command.go:102 +0x7e5
github.com/urfave/cli.(*App).RunAsSubcommand(0xc0002396c0, 0xc000536dc0)
	/home/runner/go/pkg/mod/github.com/urfave/cli@v1.22.10/app.go:405 +0xe07
github.com/urfave/cli.Command.startApp({{0x19598b4, 0x5}, {0x0, 0x0}, {0xc0006cb1d0, 0x1, 0x1}, {0x1974c0a, 0x13}, {0x0, ...}, ...}, ...)
	/home/runner/go/pkg/mod/github.com/urfave/cli@v1.22.10/command.go:378 +0xb58
github.com/urfave/cli.Command.Run({{0x19598b4, 0x5}, {0x0, 0x0}, {0xc0006cb1d0, 0x1, 0x1}, {0x1974c0a, 0x13}, {0x0, ...}, ...}, ...)
	/home/runner/go/pkg/mod/github.com/urfave/cli@v1.22.10/command.go:102 +0x7e5
github.com/urfave/cli.(*App).Run(0xc000239340, {0xc00003e0a0, 0x5, 0x5})
	/home/runner/go/pkg/mod/github.com/urfave/cli@v1.22.10/app.go:277 +0xb27
main.main()
	/home/runner/work/docker-builds/docker-builds/tctl/cmd/tctl/main.go:47 +0xa5

Steps to Reproduce the Problem

1. Deploy temporal on AKS cluster using modified chart
2. Create proper groups, users and service principals (I followed medium article with small tweaks, since it wasn't 100% up-to-date)
3. Login into admintools pod
4. Create a new namespace using UI session temporal operator namespace create --namespace <namespace> --grpc-meta=Authorization='Bearer <token_from_ui>' (works)
5. Run any of admin x-search-attributes command tctl --namespace=devl cluster get-search-attributes --auth='Bearer <token_from_ui>' or temporal operator search-attribute create --name email --type Keyword --grpc-meta=Authorization='Bearer <token_from_ui>'

Specifications

• Version: 1.24.2 - after upgrade to 1.25.1 problem still exists
• Chart version: 0.44.0 (modified to enable internal frontend and oAuth) - after upgrade to 0.50.0 problem still exists

Question

Do I miss something related to role assignment? I searched frontend, admintools and worker logs, but I couldn't find anything which might help me to debug this problem.

Update: See my comments below. Looks like a bug. Claims are not passed for DescribeNamespace and UpdateNamespace endpoints which are executed by *SearchAttributesSQL functions. This is the source of the error.

Start with the AdminGetSearchAttributes path and the *SearchAttributesSQL functions, then trace the DescribeNamespace and UpdateNamespace calls named in the update. Compare their authorization metadata with the working non-admin search-attributes command and reproduce using the documented tctl and temporal operator commands. Done means SQL search-attribute commands work with the supplied claims while preserving unauthorized behavior elsewhere.