"SSO Lifecycle" tests frequent flakes due to broker not responding with cf api info url correctly

The error

The failure we see in test output is:

  [FAILED] Expected
      <int>: 3
  to match exit code:
      <int>: 0
  In [BeforeEach] at: /go/src/github.com/cloudfoundry/cf-acceptance-tests/helpers/services/sso.go:46

Which corresponds to a non-zero exit code when curling the api info url. However, this is misleading. When we print out what the api info url is:

fmt.Println("apiInfoEndpoint is: ", apiInfoEndpoint)

right before a failed curl. We see that the value of apiInfoEndpoint is:

apiInfoEndpoint is:  http://{
    "error": true,
    "message": "undefined method `empty?' for nil:NilClass",
    "path": "http://cats-1-brkr-0007082c607e17d8.uaa-acceptance.cf-app.com/cf_api_info_url",
    "timestamp": "2024-01-02 15:56:42 +0000",
    "type": "500",
    "backtrace": [
     ...

As a comparison, right before a successful curl. The value of apiInfoEndpoint is: http://api.xxx.cf-app.com/v2/info.

This shows that an error from earlier code path was not caught (this test "debug-ability" issue is captured separately in https://github.com/cloudfoundry/cf-acceptance-tests/issues/1017).

Hence, ServiceBroker.GetApiInfoUrl is sometimes returning an error.

Flake rate

Setting --flake-attempts=2 to ./bin/test results in this test almost always failing, setting --flake-attempts=5 is what makes the test pass for us, currently.

Possibility of this as an environmental issue

This flake seems to occur frequently in our new concourse instance (https://bosh.ci.cloudfoundry.org/), whereas I can't seem to reproduce this in my local machine cats run, or with our old concourse instance. The target test subject cf deployment, and the cats config json, remain the same. We are confused of why where the test is run would affect whether ServiceBroker.GetApiInfoUrl, which is a curl to an external endpoint, flakes or not.

Start with helpers/services/sso.go around line 46 and helpers/services/broker.go at ServiceBroker.GetApiInfoUrl around line 129. Run the SSO Lifecycle test with the reported flake attempts and inspect the curl response and error path. Done means the broker API info lookup no longer causes this intermittent non-zero curl failure, with the relevant test behavior verified.