Temporal 1.24.1 <- 1.23.1 upgrade error

Expected Behavior

Application should work normally after upgrading

Actual Behavior

Application is crashing after upgrading to version 1.24.1

Steps to Reproduce the Problem

postgreSQL Schema upgrade to PostgreSQL schema v1.12 from v1.11 is successfully done.
Application upgrade to version 1.24.1 from older version was successful .

Our setup is we are using single db schema (core) and not using visibility schema but there are few tables from visibility schema in our core db.
As per logs error seems to be it is hitting visibility schema tables but not sure what is causing this ?

Specifications

• Version: 1.24.1 with postgresSQL
• Platform: AWS EKS version 1.29

temporal-frontend logs-

{"level":"error","ts":"2024-11-20T06:57:21.362Z","msg":"service failures","operation":"CountWorkflowExecutions","wf-namespace":"default","error":"CountWorkflowExecutions operation failed. Query failed: pq: column \"temporalnamespacedivision\" does not exist","logging-call-at":"telemetry.go:411","stacktrace │ │ ":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).handleError\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/te │ │ lemetry.go:411\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).UnaryIntercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/telemetry.go:202\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/ │ │ server.go:1186\ngo.temporal.io/server/service/frontend.(*RedirectionInterceptor).handleRedirectAPIInvocation.func2\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/frontend/redirection_interceptor.go:245\ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect │ │ \n\t/home/runner/work/docker-builds/docker-builds/temporal/service/frontend/dc_redirection_policy.go:125\ngo.temporal.io/server/service/frontend.(*RedirectionInterceptor).handleRedirectAPIInvocation\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/frontend/redirection_interceptor.go:242\ng │ │ o.temporal.io/server/service/frontend.(*RedirectionInterceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/frontend/redirection_interceptor.go:202\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo │ │ .temporal.io/server/common/authorization.(*Interceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/authorization/interceptor.go:181\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.temporal.io/ser │ │ ver/service/frontend.GrpcServerOptionsProvider.NewServerMetricsContextInjectorInterceptor.func2\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo │ │ .opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.51.0/interceptor.go:315\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/ │ │ mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/namespace_logger.go:85\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/ │ │ pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).NamespaceValidateIntercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/namespace_validator.go:135\ngoogle.golang.org/grpc.getChainUnaryHa │ │ ndler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.temporal.io/server/common/utf8validator.(*Validator).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/utf8validator/validate.go:182\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/ │ │ runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1186\ngo.temporal.io/server/service/frontend.GrpcServerOptionsProvider.NewServiceErrorInterceptor.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/grpc.go:178\ngoogle.golang.org/grpc.NewServer.chainUnaryServerInterceptors. │ │ chainUnaryInterceptors.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1177\ngo.temporal.io/api/workflowservice/v1._WorkflowService_CountWorkflowExecutions_Handler\n\t/home/runner/go/pkg/mod/go.temporal.io/api@v1.32.1/workflowservice/v1/service_grpc.pb.go:2217\ngoogle.golang.org/g │ │ rpc.(*Server).processUnaryRPC\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1369\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1780\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/home/runner/go/pkg/mod/go │ │ ogle.golang.org/grpc@v1.63.2/server.go:1019"}

Start with the CountWorkflowExecutions failure in the temporal-frontend logs and compare the PostgreSQL schema upgrade from v1.11 to v1.12, including the single core schema and leftover visibility tables. Trace which schema or table is queried for temporalnamespacedivision; done means the upgrade completes and CountWorkflowExecutions no longer fails.