Cannot reuse ProjectionOperation inside $lookup pipeline [DATAMONGO-2615]

Daniel Theuke opened DATAMONGO-2615 and commented

If I try to use a ProjectionOperation with andExpression("$foo.bar") inside a custom $lookup document I get the following error:

java.lang.IllegalArgumentException: Invalid reference '$meta.name'!
	at org.springframework.data.mongodb.core.aggregation.ExposedFieldsAggregationOperationContext.getReference(ExposedFieldsAggregationOperationContext.java:114) ~[spring-data-mongodb-3.0.3.RELEASE.jar:3.0.3.RELEASE]
	at org.springframework.data.mongodb.core.aggregation.ExposedFieldsAggregationOperationContext.getReference(ExposedFieldsAggregationOperationContext.java:86) ~[spring-data-mongodb-3.0.3.RELEASE.jar:3.0.3.RELEASE]
	at org.springframework.data.mongodb.core.aggregation.AggregationExpressionTransformer$AggregationExpressionTransformationContext.getFieldReference(AggregationExpressionTransformer.java:82) ~[spring-data-mongodb-3.0.3.RELEASE.jar:3.0.3.RELEASE]
	at org.springframework.data.mongodb.core.aggregation.SpelExpressionTransformer$CompoundExpressionNodeConversion.convert(SpelExpressionTransformer.java:541) ~[spring-data-mongodb-3.0.3.RELEASE.jar:3.0.3.RELEASE]
	at org.springframework.data.mongodb.core.aggregation.SpelExpressionTransformer.transform(SpelExpressionTransformer.java:113) ~[spring-data-mongodb-3.0.3.RELEASE.jar:3.0.3.RELEASE]
	at org.springframework.data.mongodb.core.aggregation.SpelExpressionTransformer.transform(SpelExpressionTransformer.java:105) ~[spring-data-mongodb-3.0.3.RELEASE.jar:3.0.3.RELEASE]
	at org.springframework.data.mongodb.core.aggregation.ProjectionOperation$ExpressionProjectionOperationBuilder$ExpressionProjection.toMongoExpression(ProjectionOperation.java:438) ~[spring-data-mongodb-3.0.3.RELEASE.jar:3.0.3.RELEASE]
	at org.springframework.data.mongodb.core.aggregation.ProjectionOperation$ExpressionProjectionOperationBuilder$ExpressionProjection.toDocument(ProjectionOperation.java:433) ~[spring-data-mongodb-3.0.3.RELEASE.jar:3.0.3.RELEASE]
	at org.springframework.data.mongodb.core.aggregation.ProjectionOperation.toDocument(ProjectionOperation.java:261) ~[spring-data-mongodb-3.0.3.RELEASE.jar:3.0.3.RELEASE]

Code:

private static final AggregationOperation EXTRACT_ELEM = replaceRoot(ElemContainer._data);
private static final ProjectionOperation ELEM_AS_METADATA = Aggregation.project()
            .andInclude("id")
            .andInclude("type")
            .andExpression("$meta.name").as("name");
    private static final ProjectionOperation ELEM_AS_NESTED_METADATA = ELEM_AS_METADATA
            // Child ids
            .and(VariableOperators.mapItemsOf(
                    ObjectOperators.valueOf("childIds").toArray())
                    .as("this")
                    .andApply(ctx -> new Document()
                            .append("type", "$$this.k")
                            .append("id", "$$this.v")))
            .as("children"); // Named children to be overwritten later on

    private static final AggregationOperation LOOKUP_CHILDREN = ctx -> new Document()
        .append("$lookup", new Document()
            .append("from", "configuration")
            .append("let", new Document()
                    .append("childIds", "$children"))
            .append("pipeline", asList(
                    new Document() // Join condition
                            .append("$match", new Document()
                                    .append("$expr", new Document()
                                            .append("$in", asList("$_id", "$$childIds")))),
                    EXTRACT_ELEM,
                    ELEM_AS_METADATA.toDocument(ctx.continueOnMissingFieldReference()))) // <-- BOOM
            .append("as", "children"); // Overwrite "children" field

// --------------------------

final TypedAggregation<ElemContainer> aggregation = newAggregate(
                match(criteria),
                EXTRACT_ELEM,
                ELEM_AS_NESTED_METADATA,
                LOOKUP_CHILDREN);
        return this.mongoTemplate.aggregate(aggregation, NestedElementMetadata.class).getMappedResults();

In this case the continueOnMissingFieldReference() doesn't seem to work at all.
(ExposedFieldsAggregationOperationContext doesn't implement that feature at all)

The default implementation should probably be rewritten to return a delegating variant that wraps the related call similar to RelaxedTypeBasedAggregationOperationContext.

Or is there another way to create a nested context?

Affects: 3.0.3 (Neumann SR3)