Archival does not show list of archived workflows

Expected Behavior

Viewing the Archival tab for a namespace which has archived workflows should display them.

Actual Behavior

The Archival tab is displaying the message "No Workflows running in this Namespace" even though there are archived workflows present in the configured archival S3 bucket. The GET call to api/v1/namespaces/<namespace>/archived-workflows?query= is failing with an HTTP 400.

Steps to Reproduce the Problem

1. Create an S3 bucket for Archival.
2. Ensure that Temporal's K8s service account has the below IAM permissions, changing the bucket name here to match.

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::temporal-workflow-archive-bucket",
                "arn:aws:s3:::temporal-workflow-archive-bucket/*"
            ]
        }
    ]
}

4. Add the below configuration to the Temporal Helm chart, changing the bucket name here to match.

config:
    archival:
      history:
        state: enabled
        enableRead: true
        provider:
          s3store:
            region: <aws region>
      visibility:
        state: enabled
        enableRead: true
        provider: 
          s3store:
            region: <aws region>

    namespaceDefaults:
      archival:
        history:
          state: enabled
          URI: s3://temporal-workflow-archive-bucket
        visibility:
          state: enabled
          URI: s3://temporal-workflow-archive-bucket

5. Turn on archival and visibility for a namespace that has workflows running, and wait for the retention period.
6. View the archival bucket and observe that there are files present for those archived workflows.
7. Go to the Archival tab for that namespace and observe the message "No Workflows running in this Namespace" and the failed call to api/v1/namespaces/<namespace>/archived-workflows?query=.

Specifications

• Version: 1.22.5
• Platform: AWS, EKS, amd64

Start with the GET endpoint api/v1/namespaces//archived-workflows?query= and reproduce the HTTP 400 using the S3 archival and visibility configuration described. Trace the request through the archival workflow listing path; done means the endpoint succeeds and the Archival tab lists workflows present in the configured bucket.