[BUG] Error parsing search response aggregation, property name 'doc_count_error_upper_bound' is not in the 'type#name' format. Make sure the request has 'typed_keys' set.

What is the bug?

The java client can't parse an aggregation with show_term_doc_count_error
I already configured the SERIALIZE_TYPED_KEYS to false.

jakarta.json.stream.JsonParsingException: Property name 'doc_count_error_upper_bound' is not in the 'type#name' format. Make sure the request has 'typed_keys' set.
	at org.opensearch.client.json.ExternallyTaggedUnion$TypedKeysDeserializer.deserializeEntry(ExternallyTaggedUnion.java:129)
	at org.opensearch.client.opensearch._types.aggregations.MultiBucketBase.lambda$setupMultiBucketBaseDeserializer$0(MultiBucketBase.java:159)
	at org.opensearch.client.json.ObjectDeserializer.parseUnknownField(ObjectDeserializer.java:214)
	at org.opensearch.client.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:183)
	at org.opensearch.client.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:146)
	at org.opensearch.client.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:97)
	at org.opensearch.client.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:60)
	at org.opensearch.client.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:343)
	at org.opensearch.client.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:308)
	at org.opensearch.client.json.UnionDeserializer$SingleMemberHandler.deserialize(UnionDeserializer.java:91)
	at org.opensearch.client.json.UnionDeserializer.deserialize(UnionDeserializer.java:331)
	at org.opensearch.client.json.UnionDeserializer.deserialize(UnionDeserializer.java:285)
	at org.opensearch.client.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:81)
	at org.opensearch.client.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:185)
	at org.opensearch.client.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:146)
	at org.opensearch.client.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:97)
	at org.opensearch.client.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:60)
	at org.opensearch.client.json.ExternallyTaggedUnion$Deserializer.deserialize(ExternallyTaggedUnion.java:94)
	at org.opensearch.client.json.ExternallyTaggedUnion$TypedKeysDeserializer.deserializeEntry(ExternallyTaggedUnion.java:136)
	at org.opensearch.client.json.ExternallyTaggedUnion$TypedKeysDeserializer.deserialize(ExternallyTaggedUnion.java:119)
	at org.opensearch.client.json.ExternallyTaggedUnion$TypedKeysDeserializer.deserialize(ExternallyTaggedUnion.java:106)
	at org.opensearch.client.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:87)
	at org.opensearch.client.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:81)
	at org.opensearch.client.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:185)
	at org.opensearch.client.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:146)
	at org.opensearch.client.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:87)
	at org.opensearch.client.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:91)
	at org.opensearch.client.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:55)
	at org.opensearch.client.transport.endpoints.EndpointWithResponseMapperAttr$1.deserialize(EndpointWithResponseMapperAttr.java:68)
	at org.opensearch.client.transport.rest_client.RestClientTransport.decodeResponse(RestClientTransport.java:360)
	at org.opensearch.client.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:326)
	at org.opensearch.client.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:148)
	at org.opensearch.client.opensearch.OpenSearchClient.search(OpenSearchClient.java:1378)

Thanks in advance for your work on Opensearch !

How can one reproduce the bug?

Request

{
  "aggregations": {
    "country": {
      "meta": {
        "termType": "STRING"
      },
      "terms": {
        "field": "country.$keyword",
        "order": [
          {
            "_key": "asc"
          },
          {
            "_count": "desc"
          }
        ],
        "show_term_doc_count_error": true,
        "size": 5
      }
    },
    "countryNumberOfDistinctValues": {
      "cardinality": {
        "field": "country.$keyword",
        "precision_threshold": 3000
      }
    }
  },
  "from": 0,
  "size": 20,
  "sort": [
    {
      "_score": {
        "order": "desc"
      }
    }
  ]
}

Response

{
  "took": 53,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "max_score": 0.47589022,
    "hits": [
      {
        "_index": "nodes_discovery",
        "_id": "id1",
        "_score": 0.47589022,
        "_source": {
          "country": "PR China"
        }
      },
      {
        "_index": "nodes_discovery",
        "_id": "id2",
        "_score": 0.29555285,
        "_source": {
          "country": "Taiwan"
        }
      }
    ]
  },
  "aggregations": {
    "cardinality#countryNumberOfDistinctValues": {
      "value": 2
    },
    "sterms#country": {
      "meta": {
        "termType": "STRING"
      },
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "pr china",
          "doc_count": 1,
          "doc_count_error_upper_bound": 0
        },
        {
          "key": "taiwan",
          "doc_count": 1,
          "doc_count_error_upper_bound": 0
        }
      ]
    }
  }
}

After debugging, the error is during parsing this block

{
  "key": "pr china",
  "doc_count": 1,
  "doc_count_error_upper_bound": 0
}

Java client configuration

  public static final JsonpMapper OPENSEARCH_JMAPPER =
      // https://github.com/opensearch-project/opensearch-java/issues/741
      new JacksonJsonpMapper(new ObjectMapper()).withAttribute(SERIALIZE_TYPED_KEYS, false);
      
  @Bean
  @Qualifier("openSearchClient")
  public OpenSearchClient opensearchClient() {
    OpenSearchTransport transport = new RestClientTransport(getRestClient(), OPENSEARCH_JMAPPER);
    return new OpenSearchClient(transport);
  }

  private RestClient getRestClient() {
    HttpHost[] httpHosts = urls.stream().map(HttpHost::create).toArray(HttpHost[]::new);
    RestClientBuilder clientBuilder = RestClient.builder(httpHosts).setCompressionEnabled(true);
    if (username.isPresent() && password.isPresent()) {
      clientBuilder.setHttpClientConfigCallback(
          addBasicCredentialsProvider(username.get(), password.get()));
    }

    clientBuilder.setRequestConfigCallback(
        requestConfigBuilder ->
            requestConfigBuilder
                .setSocketTimeout(socketTimeout)
                .setConnectTimeout(connectTimeout)
                .setContentCompressionEnabled(true));
    return clientBuilder.build();
  }

What is the expected behavior?

No error

What is your host/environment?

Opensearch 2.19.3
Opensearch java client 2.26.0
Java 21

Do you have any additional context?

I'm currently migrate from the high level java client.
With the old client the parsing of the exact same response worked well.

Start with ExternallyTaggedUnion.java and MultiBucketBase.java, following the stack trace through typed-key deserialization for the supplied aggregation response. Reproduce the show_term_doc_count_error case and verify that the response, including doc_count_error_upper_bound in buckets, parses without an exception.