dotnet / dotnet/efcore

GroupBy aggregate over a required navigation with a query filter drops entire groups (Regression in 11.0)

Open
#38,965 3 comments 0 reactions 1 assignee Claimed by @AndriySvyryd View on GitHub
area-groupby area-query customer-reported regression
Dominant language
C#
Stars
14.8k
Forks
3.4k
PR merge metrics
PR metrics pending

Description

### Bug description

When an aggregate over a grouping reaches through a required navigation whose principal has a query filter, the group is silently removed from the results — including its `Count()`, which never touched the navigation.

Lifting the aggregate into a pre-`GroupBy` join (#38668) moved the navigation from a correlated subquery in the projection to an `INNER JOIN` beneath the `GROUP BY`. A correlated subquery cannot remove a source row; it returns `NULL` for it. An `INNER JOIN` onto a filtered subquery can, and does — so a row whose principal is filtered out no longer reaches the grouping, and if that was the group's only row, the group itself never forms.

Northwind with the natural soft-delete filter on products:

```csharp
modelBuilder.Entity().HasQueryFilter(p => !p.Discontinued);
```

Order 10248 has one line, for a live product. Order 10249 has one line, for a discontinued one.

```csharp
// 1 — Count alone
ctx.OrderDetails.GroupBy(od => od.OrderID)
.Select(g => new { g.Key, Count = g.Count() })

// 2 — Count plus an aggregate that traverses Product
ctx.OrderDetails.GroupBy(od => od.OrderID)
.Select(g => new { g.Key, Count = g.Count(), MaxStock = g.Max(x => (int?)x.Product.UnitsInStock) })

// 3 — the same question, traversal moved into the GroupBy element selector
ctx.OrderDetails.GroupBy(od => od.OrderID, od => new { od.Product.UnitsInStock, od.Quantity })
.Select(g => new { g.Key, Count = g.Count(), MaxStock = g.Max(x => (int?)x.UnitsInStock) })
```

| | 10.0 (before #38668) | 11.0 rc1 / main |
| --- | --- | --- |
| 1 | 2 groups — `10248:Count=1`, `10249:Count=1` | 2 groups — unchanged |
| 2 | 2 groups — `10248:Count=1,Max=39`, `10249:Count=1,Max=null` | **1 group** — `10248:Count=1,Max=39`; **order 10249 is gone** |
| 3 | 2 groups — same as 10.0 query 2 | 2 groups — unchanged |

Two things stand out beyond the regression itself.

**`Count` changes even though it traverses nothing.** Queries 1 and 2 differ only by the presence of a second, unrelated aggregate, and adding it removes a group from the result. Whether order 10249 appears is decided by what a *sibling* aggregate in the same projection does.

**Queries 2 and 3 ask the same question and now disagree.** The only difference is whether `od.Product` is reached from the aggregate lambda or from the element selector, which selects a different translation. Before #38668 they agreed.

### Expected behaviour

Query 2 returns both orders, with `MaxStock = null` for order 10249 — the 10.0 behaviour, and what query 3 still returns today. An aggregate materializes no `Product`; it reads `UnitsInStock` and folds it. A filtered-out principal should make the aggregate empty, not delete the order.

### SQL

Before #38668 — the traversal is isolated in a correlated subquery, so `COUNT(*)` sees every line:

```sql
SELECT "o"."OrderID" AS "Key", COUNT(*) AS "Count", (
SELECT MAX("p0"."UnitsInStock")
FROM "OrderDetails" AS "o0"
INNER JOIN (
SELECT "p"."ProductID", "p"."UnitsInStock"
FROM "Products" AS "p"
WHERE NOT ("p"."Discontinued")
) AS "p0" ON "o0"."ProductID" = "p0"."ProductID"
WHERE "o"."OrderID" = "o0"."OrderID") AS "MaxStock"
FROM "OrderDetails" AS "o"
GROUP BY "o"."OrderID"
```

11.0 rc1 / main — the filter is now a join condition beneath the `GROUP BY`, so it removes rows from the grouping:

```sql
SELECT "o"."OrderID" AS "Key", COUNT(*) AS "Count", MAX("p"."UnitsInStock") AS "MaxStock"
FROM "OrderDetails" AS "o"
INNER JOIN "Products" AS "p" ON "o"."ProductID" = "p"."ProductID" AND NOT ("p"."Discontinued")
GROUP BY "o"."OrderID"
```

The same plan shape is generated on SQL Server.

### Relationship to #19801

I'm aware that removing a dependent whose required principal is filtered out is deliberate, and that `PossibleIncorrectRequiredNavigationWithQueryFilterInteractionWarning` exists for exactly this model. I don't think that covers this case:

- The rationale in #19801 is referential integrity — a required principal that isn't there can't be materialized. Nothing is materialized here; the aggregate reads a column.
- It doesn't explain why `Count()`, which never traverses the navigation, changes its answer.
- It doesn't explain why queries 2 and 3 disagree, nor why query 2's behaviour changed in 11.0 while query 3's did not.
- The suggested remedy — matching filters on both entities — changes the intent, since it also removes those lines from `Count`. It cannot express "count every line, and report the stock figure as unavailable", which is what 10.0 returned.

### Your code

```csharp
using Microsoft.EntityFrameworkCore;

using var ctx = new NorthwindContext();
ctx.Database.EnsureCreated();

ctx.AddRange(
new Product { ProductID = 1, ProductName = "Chai", Discontinued = false, UnitsInStock = 39 },
new Product { ProductID = 2, ProductName = "Mishi Kobe Niku", Discontinued = true, UnitsInStock = 29 },
new Order { OrderID = 10248 },
new Order { OrderID = 10249 },
new OrderDetail { OrderID = 10248, ProductID = 1, Quantity = 12 },
new OrderDetail { OrderID = 10249, ProductID = 2, Quantity = 9 });
ctx.SaveChanges();
ctx.ChangeTracker.Clear();

var q1 = ctx.OrderDetails.GroupBy(od => od.OrderID)
.Select(g => new { g.Key, Count = g.Count() })
.OrderBy(x => x.Key).ToList();

var q2 = ctx.OrderDetails.GroupBy(od => od.OrderID)
.Select(g => new { g.Key, Count = g.Count(), MaxStock = g.Max(x => (int?)x.Product.UnitsInStock) })
.OrderBy(x => x.Key).ToList();

var q3 = ctx.OrderDetails.GroupBy(od => od.OrderID, od => new { od.Product.UnitsInStock, od.Quantity })
.Select(g => new { g.Key, Count = g.Count(), MaxStock = g.Max(x => (int?)x.UnitsInStock) })
.OrderBy(x => x.Key).ToList();

Console.WriteLine($"1) {q1.Count} groups"); // 2
Console.WriteLine($"2) {q2.Count} groups"); // 1 on 11.0, 2 on 10.0
Console.WriteLine($"3) {q3.Count} groups"); // 2

public class Product
{
public int ProductID { get; set; }
public string ProductName { get; set; } = null!;
public bool Discontinued { get; set; }
public short UnitsInStock { get; set; }
}

public class Order
{
public int OrderID { get; set; }
}

public class OrderDetail
{
public int OrderID { get; set; }
public int ProductID { get; set; }
public short Quantity { get; set; }
public Order Order { get; set; } = null!;
public Product Product { get; set; } = null!;
}

public class NorthwindContext : DbContext
{
private readonly Microsoft.Data.Sqlite.SqliteConnection _connection = new("DataSource=:memory:");

public DbSet Products => Set();
public DbSet Orders => Set();
public DbSet OrderDetails => Set();

public NorthwindContext() => _connection.Open();

protected override void OnConfiguring(DbContextOptionsBuilder b)
=> b.UseSqlite(_connection);

protected override void OnModelCreating(ModelBuilder mb)
{
mb.Entity().HasQueryFilter(p => !p.Discontinued);

mb.Entity().HasKey(od => new { od.OrderID, od.ProductID });
mb.Entity().HasOne(od => od.Order).WithMany().HasForeignKey(od => od.OrderID);
mb.Entity().HasOne(od => od.Product).WithMany().HasForeignKey(od => od.ProductID);
}

public override void Dispose()
{
base.Dispose();
_connection.Dispose();
}
}
```

### Stack traces

```text

```

### Verbose output

```text

```

### EF Core version

11.0.0-rc.1

### Database provider

_No response_

### Target framework

_No response_

### Operating system

_No response_

### IDE

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.