JSON-absent required complex collection materializes as null under ComplexProperty().ToJson(), then throws InvalidOperationException in PrepareToSave on save
- Dominant language
- C#
- Stars
- 14.8k
- Forks
- 3.4k
- PR merge metrics
- PR metrics pending
Description
### Bug description
When a complex type is mapped to a JSON column via `ComplexProperty().ToJson()`, and the stored JSON is **missing the key for a nested complex collection** (e.g. a row written before that collection property was added — schema evolution), EF Core **materializes that collection as `null`** rather than running the CLR initializer / producing an empty collection.
This has two downstream consequences:
1. **Read hazard.** Any consumer that touches the collection (`.Add`, iterate, `.Count`) throws `NullReferenceException`. The CLR property initializer (`= []`) is ignored during materialization.
2. **Save hazard.** Loading such a row **tracked**, mutating anything on the entity, and calling `SaveChanges()` throws `InvalidOperationException` from `PrepareToSave.CheckForNullComplexProperties` (see stack trace below).
The insert path is unaffected — only **pre-existing rows whose stored JSON predates the collection field** are poison-on-touch, which makes this a classic schema-evolution failure that only surfaces in production after a deploy.
**Expected behavior:** a required complex collection that is absent from the stored JSON should materialize as an **empty collection** (consistent with the CLR initializer and with how `System.Text.Json` POCO deserialization behaves for an absent array property), so that reads return `[]` and `SaveChanges()` does not throw for a collection that is legitimately empty. At minimum, a row that round-trips (load → save) should not become un-saveable purely because the collection key was absent in the stored JSON.
**Workaround we're using:** coalesce the null complex collections to empty after materialization (via the `ChangeTracker.Tracked` event / a post-materialization pass) before the entity is read by consumers or saved.
**Related:** #37162 (milestone 10.0.1) and #37890 (milestone 10.0.6) fix adjacent complex-type-null cases but do not cover the absent-collection case — this still reproduces on 10.0.10. Also related: #37224 (dup of #37162), #36402.
### Your code
```csharp
// net10.0 console app; Npgsql.EntityFrameworkCore.PostgreSQL 10.0.0 (EF Core 10.0.2; also reproduces on 10.0.10).
// Needs a local PostgreSQL (any recent version).
using Microsoft.EntityFrameworkCore;
await using (var ctx = new AppDbContext()) {
await ctx.Database.EnsureDeletedAsync();
await ctx.Database.EnsureCreatedAsync();
// 1. Insert a row with the CURRENT model shape.
ctx.Add(new Order {
Id = Guid.NewGuid(),
Data = new OrderDetails {
Shipments = [new Shipment { Carrier = "A", Parcels = [new Parcel { TrackingCode = "p1" }] }],
},
});
await ctx.SaveChangesAsync();
// 2. Simulate an OLD-SHAPE row: strip the key of a later-added nested collection
// (i.e. a row written before Parcel.InspectionTagIds existed).
await ctx.Database.ExecuteSqlRawAsync(
"UPDATE orders SET data = data #- '{Shipments,0,Parcels,0,InspectionTagIds}'");
}
await using (var ctx = new AppDbContext()) {
// 3. READ hazard: the JSON-absent collection materializes as null (expected: []).
var order = await ctx.Orders.AsNoTracking().SingleAsync();
Console.WriteLine(order.Data.Shipments[0].Parcels[0].InspectionTagIds is null); // True
}
await using (var ctx = new AppDbContext()) {
// 4. SAVE hazard: tracked load + any mutation + SaveChanges throws InvalidOperationException
// in PrepareToSave.CheckForNullComplexProperties.
var order = await ctx.Orders.SingleAsync();
order.Data.Shipments[0].Parcels.Add(new Parcel { TrackingCode = "p2", InspectionTagIds = [] });
await ctx.SaveChangesAsync(); // throws
}
public class AppDbContext : DbContext {
public DbSet Orders => Set();
protected override void OnConfiguring(DbContextOptionsBuilder options)
=> options.UseNpgsql("Host=localhost;Port=5432;Database=efrepro;Username=postgres;Password=postgres");
protected override void OnModelCreating(ModelBuilder modelBuilder)
=> modelBuilder.Entity(e => {
e.ToTable("orders");
e.HasKey(x => x.Id);
e.ComplexProperty(x => x.Data, d => d.ToJson("data"));
});
}
public class Order {
public Guid Id { get; set; }
public OrderDetails Data { get; set; } = new();
}
public class OrderDetails {
public List Shipments { get; set; } = [];
}
public class Shipment {
public string Carrier { get; set; } = "";
public List Parcels { get; set; } = [];
}
public class Parcel {
public string TrackingCode { get; set; } = "";
public List InspectionTagIds { get; set; } = []; // the later-added field, absent from old rows
}
```
### Stack traces
```text
System.InvalidOperationException: The complex type property '.' is configured as required (non-nullable) but has a null value when saving changes.
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntryBase.g__CheckForNullComplexProperties|99_2(<>c__DisplayClass99_0&)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntryBase.PrepareToSave()
(property name matches whichever required complex collection materialized as null from the JSON-absent key)
```
### Verbose output
_No response_
### EF Core version
10.0.2 (reproduces unchanged on 10.0.10 — every released 10.0.x patch)
### Database provider
Npgsql.EntityFrameworkCore.PostgreSQL 10.0.0
### Target framework
.NET 10.0
### Operating system
macOS (Apple Silicon); PostgreSQL in Docker
### IDE
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.