opensearch-project / opensearch-project/opensearch-java

Composite Aggregation after key ignored

Open
#785 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
165
Forks
250
Avg merge
1d 18h
Merged PRs (30d)
26

Description

What is the bug?

when searching with composite aggregation the supplied afterKey is ignored and we always get back the same initial afterKey.
Therefore, iterating with composite aggregation until the afterKey is null, leads to an infinite loop

client version

we encountered this bug with client version 2.6.0, 2.70 and also after upgrading to the latest 2.8.1 client version

How can one reproduce the bug?

search with composite aggregation in a loop until the afterKey is null.
Since the afterKey is ignored, it will always search from the beginning and repeatedly return the same afterKey

What is the expected behavior?

the afterKey should should be considered, so that after each search we get back the "next" afterKey

workaround

Unfortunately we switched back to the rest client :(

Code sample to reproduce infinite loop
List<Map<String, CompositeAggregationSource>> compositeAggSourceList = new ArrayList<>();

compositeAggSourceList.add(Collections.singletonMap("tenant_id", new CompositeAggregationSource.Builder()
				.terms(new TermsAggregation.Builder()
						.field("tenant_id")
						.build())
		.build()));
compositeAggSourceList.add(Collections.singletonMap("user_id", new CompositeAggregationSource.Builder()
		.terms(new TermsAggregation.Builder()
				.field("user_id")
				.build())
		.build()));
					
CompositeAggregate compositeAggregate = null;
do {
	// update the composite aggregation after key
	Map<String, String> afterKey;
	if(compositeAggregate !=null){
		afterKey = compositeAggregate.afterKey().entrySet().stream().collect(Collectors.toMap(Map.Entry::getKey,
				e -> new String(e.getValue().toString())));
	} else {
		afterKey = null;
	}

	SearchRequest searchRequest = new SearchRequest.Builder()
			.index("example-index")
			.size(0)
			.aggregations("composite-agg", new Aggregation.Builder()
					.composite(c-> afterKey !=null ? c.sources(compositeAggSourceList).size(10).after(afterKey) : c.sources(compositeAggSourceList).size(10))
					.aggregations("incidents",
							a -> a.sum(t -> t.field("incidents")))
					.aggregations("activities",
							a -> a.sum(t -> t.field("activities")))
					.build())
			.build();

	SearchResponse<Map> searchResponse = openSearchClient.get().search(searchRequest, Map.class);

	compositeAggregate = searchResponse.aggregations().get("composite-agg").composite();
	
	// our internal logic...

} while (compositeAggregate.afterKey()!=null);

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplied Java reproduction and trace how SearchRequest builds the composite aggregation when an afterKey is present. Verify the request reaches the client correctly and reproduce the loop; done means successive searches honor the supplied afterKey and return the next page until afterKey is null.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.