Chef search results are limited to 10,000 records when using external OpenSearch 1.3.x regardless of max_result_window

Chef Server Version

Chef Infra Server 15.7.0

Platform Details

Ubuntu 20.04 running on c5.2xlarge instances in AWS EC2

Configuration

Chef HA cluster with external OpenSearch 1.3.14 and PSQL 13 backends. Recently migrated from 12.22.4 to 15.7.0.

Scenario

Since migrating to Infra Server 15.7.0, all chef search results are capped at 10,000. The max_result_window on the index is currently set to 1,000,000. We have manually adjusted it several times as a test with no result. No matter what value we configure in the index, only 10,000 results are returned.

Steps to Reproduce

Provision a Chef HA cluster with an OpenSearch 1.3.x backend, register at least 10,000 nodes, and then attempt to run knife status or knife search *:*.

Expected Result

All registered node objects should be returned.

Actual Result

Only 10,000 objects are returned by the query, regardless of how many objects are in the index. In this particular case, we have over 20,000 registered nodes but only 10,000 are returned in the search results.

$ knife node list | wc -l
   20343
   
$ knife status | wc -l
   10000

$ curl https://opensearch.host/chef/_settings -u chef_user | jq .chef.settings.index.max_result_window
"1000000"

Breadcrumbs

This appears to be related to differences in the search API between Elasticsearch 6 and OpenSearch 1.3.x. The following code assumes that the value for "total" will equal the total number of records, which was the case in elasticsearch 6.
https://github.com/chef/chef-server/blob/main/src/oc_erchef/apps/chef_index/src/chef_opensearch.erl#L84-L94

However, OpenSearch will return a value of '10000' in the total field unless ?track_total_hits=true is added to the query. For example, Chef server 15.7.0 receives the following result from the default OpenSearch query: "hits":{"total":{"value":10000,"relation":"gte"} compared to the following when track_total_hits is added: "hits":{"total":{"value":61964,"relation":"eq"}. This behavior is further documented in the following bug report:
https://github.com/opensearch-project/OpenSearch/issues/9720

As well as the following forum thread:
https://forum.opensearch.org/t/opensearch-dashboards-shows-10000-as-hits-total/8397

Start with src/oc_erchef/apps/chef_index/src/chef_opensearch.erl around lines 84-94 and inspect how the OpenSearch query handles hits.total. Reproduce with an OpenSearch 1.3.x backend using knife status or knife search :, then verify that searches return more than 10,000 records when more are indexed.