Problem with replacement of @DBRef to @DocumentReference

Hello!

I would like to describe a problem with replacement of annotation @DBRef to annotation @DocumentReference in case old data in MongoDB exist. For this purpose I have created a project with reproducing of this problem: https://github.com/ALGA0887/reference.git.

So lets imagine the following class in previous releases of some application:

@Data
@AllArgsConstructor
@Document(collection = "guests")
public class Guest {
    @Id
    private String id;
    @Field
    private String name;
    @DBRef
    private List<OrderItem> orderItems;
}

Here there is a @DBRef to the list of OrderItem:

@Data
@AllArgsConstructor
@Document(collection = "order-items")
public class OrderItem {
    @Id
    private String id;
    @Field
    private String name;
    @Field
    private List<String> guestIds;
}

OrderItem itself also contains information about guestIds.
So in previous releases guests were saved in MongoDB with DBRef to orderItems.

In current release it was decided not to have a field orderItems in guests collection since it is overhead. And it was planned to change the annotation @DBRef to annotation @DocumentReference in the following way:

@Data
@Document(collection = "guests")
public class Guest {
    @Id
    private String id;
    @Field
    private String name;
    @ReadOnlyProperty
    @DocumentReference(lookup = "{'guestIds':?#{#self._id}}")
    private List<OrderItem> orderItems;
    public Guest(String id, String name) {
        this.id = id;
        this.name = name;
    }
}

For new records in guests such changes work as expected (test com.alga.reference.ReferenceApplicationTests#testLoadGuestWithoutDbRef). But old records can't be read with the following exception:

org.springframework.expression.spel.SpelEvaluationException: EL1008E: Property or field '_id' cannot be found on object of type 'com.mongodb.DBRef' - maybe not public or not valid?

	at org.springframework.expression.spel.ast.PropertyOrFieldReference.readProperty(PropertyOrFieldReference.java:228)
	at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:111)
	at org.springframework.expression.spel.ast.PropertyOrFieldReference$AccessorValueRef.getValue(PropertyOrFieldReference.java:416)
	at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:98)
	at org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:119)
	at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:309)
	at org.springframework.data.mongodb.util.json.EvaluationContextExpressionEvaluator.evaluateExpression(EvaluationContextExpressionEvaluator.java:69)
	at org.springframework.data.mongodb.util.json.ParameterBindingContext.evaluateExpression(ParameterBindingContext.java:115)
	at org.springframework.data.mongodb.util.json.ParameterBindingJsonReader.evaluateExpression(ParameterBindingJsonReader.java:535)
	at org.springframework.data.mongodb.util.json.ParameterBindingJsonReader.bindableValueFor(ParameterBindingJsonReader.java:395)
	at org.springframework.data.mongodb.util.json.ParameterBindingJsonReader.readBsonType(ParameterBindingJsonReader.java:300)
	at org.springframework.data.mongodb.util.json.ParameterBindingDocumentCodec.decode(ParameterBindingDocumentCodec.java:237)
	at org.springframework.data.mongodb.util.json.ParameterBindingDocumentCodec.decode(ParameterBindingDocumentCodec.java:182)
	at org.springframework.data.mongodb.core.convert.ReferenceLookupDelegate.computeFilter(ReferenceLookupDelegate.java:281)
	at org.springframework.data.mongodb.core.convert.ReferenceLookupDelegate.readReference(ReferenceLookupDelegate.java:109)
	at org.springframework.data.mongodb.core.convert.DefaultReferenceResolver.resolveReference(DefaultReferenceResolver.java:76)
	at org.springframework.data.mongodb.core.convert.MappingMongoConverter.readAssociation(MappingMongoConverter.java:655)
...

So my question is why DBRef is taken into account in lookup expression? Is it bug in reference resolving?

In my project I have tried the following workaround and it works:

@Configuration
public class MongoConfig {
    @Bean
    @Profile("workaround")
    public MappingMongoConverter mappingMongoConverter(MongoDatabaseFactory factory, MongoMappingContext mappingContext) {
        // GuestDbRefResolver is a workaround for loading of old data: guests with DBRef-s.
        // With DefaultDbRefResolver loading of old data is failed with SpelEvaluationException and the following message:
        //      "EL1008E: Property or field '_id' cannot be found on object of type 'com.mongodb.DBRef' - maybe not public or not valid?".
        // To check it comment @ActiveProfiles(value = "workaround") on a test class com.alga.reference.ReferenceApplicationTests
        // and run test com.alga.reference.ReferenceApplicationTests.testLoadGuestWithDbRef
        DbRefResolver dbRefResolver = new GuestDbRefResolver(factory);
        return new MappingMongoConverter(dbRefResolver, mappingContext);
    }
}

public class GuestDbRefResolver extends DefaultDbRefResolver {
    public GuestDbRefResolver(MongoDatabaseFactory mongoDbFactory) {
        super(mongoDbFactory);
    }
    @SneakyThrows
    @Override
    public Object resolveReference(MongoPersistentProperty property, Object source, ReferenceLookupDelegate referenceLookupDelegate, MongoEntityReader entityReader) {
        Object resultSource = source;
        if (source instanceof DocumentReferenceSource drs && drs.getTargetSource() != null && "orderItems".equals(property.getFieldName())) {
            Class<?> ownerClass = property.getOwner().getTypeInformation().getType();
            Class<?> targetClass = property.getAssociationTargetType();
            if (ownerClass == Guest.class && targetClass == OrderItem.class) {
                // Why constructor for DocumentReferenceSource is not public?
                Constructor<DocumentReferenceSource> drsc = DocumentReferenceSource.class.getDeclaredConstructor(Object.class, Object.class);
                drsc.setAccessible(true);
                resultSource = drsc.newInstance(drs.getSelf(), null);
            }
        }
        return super.resolveReference(property, resultSource, referenceLookupDelegate, entityReader);
    }
}

But if we imagine that migration of data is impossible (I mean remove DBRef for old records in guests) then how we can apply such changes without any workaround at all?

To reproduce a problem just comment @ActiveProfiles(value = "workaround") in ReferenceApplicationTests and run test testLoadGuestWithDbRef.

@RunWith(SpringRunner.class)
@SpringBootTest(classes = ReferenceApplication.class)
//@ActiveProfiles(value = "workaround")
public class ReferenceApplicationTests {

    ...    

    @Test
    public void testLoadGuestWithDbRef() {
        ...
    }
}

Start by running com.alga.reference.ReferenceApplicationTests#testLoadGuestWithDbRef in the linked reproducer, with the workaround profile disabled. Read ReferenceLookupDelegate, DefaultDbRefResolver, and the stack trace around DocumentReferenceSource to understand why the old DBRef reaches the lookup expression. Done means old guests load successfully while the new @DocumentReference behavior remains intact without the custom resolver.