`highly_variable_genes` with `batch_key` and `subset=True` keeps the wrong genes when `n_top_genes` is not set

Please make sure these conditions are met

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of scanpy.
• (optional) I have confirmed this bug exists on the main branch of scanpy.

What happened?

sc.pp.highly_variable_genes(adata, batch_key=..., subset=True) with the default cutoffs (n_top_genes=None, flavors seurat and cell_ranger) keeps the wrong genes whenever var_names are not sorted alphabetically. No error, the subsetted object just contains mostly non-HVGs.

On pbmc3k with two batches, subset=False flags 1857 genes. subset=True also keeps 1857 genes, but only 269 of them are the flagged ones.

This looks like the other half of #3027. In _highly_variable_genes_batched, df.groupby(df.index).agg(...) returns the genes sorted by name. #3042 put the original order back, but only inside the if isinstance(cutoff, int): branch (df = df.loc[df_orig_ind]). The else branch for the mean/dispersion cutoffs never reorders, so adata._inplace_subset_var(df["highly_variable"]) applies a name-sorted boolean mask by position. subset=False is fine because assigning into adata.var aligns on the index. With inplace=False the returned frame is also in sorted order instead of var_names order.

test_subset_inplace_consistency only runs with n_top_genes=15, so the default branch is not covered.

Moving the reindex out of the if so both branches get it fixes it for me. Happy to open a PR with a test.

Minimal code sample

# /// script
# requires-python = ">=3.12"
# dependencies = [
#   "scanpy@git+https://github.com/scverse/scanpy.git@main",
# ]
# ///

import numpy as np
import scanpy as sc

adata = sc.datasets.pbmc3k()
adata.obs["batch"] = np.where(np.arange(adata.n_obs) % 2 == 0, "a", "b")
sc.pp.filter_genes(adata, min_cells=3)
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)

flagged = adata.copy()
sc.pp.highly_variable_genes(flagged, batch_key="batch")
expected = set(flagged.var_names[flagged.var["highly_variable"]])

subsetted = adata.copy()
sc.pp.highly_variable_genes(subsetted, batch_key="batch", subset=True)
kept = set(subsetted.var_names)

print("HVGs flagged with subset=False:", len(expected))
print("genes kept with subset=True:   ", len(kept))
print("kept genes that are HVGs:      ", len(kept & expected))
print("all kept genes flagged HVG:    ", bool(subsetted.var["highly_variable"].all()))

Error output

HVGs flagged with subset=False: 1857
genes kept with subset=True:    1857
kept genes that are HVGs:       269
all kept genes flagged HVG:     False

Same numbers on 1.12.4 from PyPI.

Versions

scanpy	1.14.0.dev8+g5f2accbe7
----	----
packaging	26.3
threadpoolctl	3.7.0
typing-inspection	0.4.4
fonttools	4.65.0
pillow	12.3.0
numpy	2.5.3
joblib	1.6.0
python-dateutil	2.9.0.post0
pandas	3.0.5
matplotlib	3.11.2
six	1.17.0
llvmlite	0.49.0
tblib	3.2.2
coverage	7.16.1
narwhals	2.26.0
numcodecs	0.16.5
scipy	1.18.1
toolz	1.1.0
pyparsing	3.3.2
numba	0.67.0
fast-array-utils	1.5
natsort	8.4.0
scverse-misc	0.1.6
legacy-api-wrap	1.5
python-dotenv	1.2.3
pyarrow	25.0.1
psutil	7.2.2
scikit-learn	1.9.1
donfig	0.8.1.post1
zarr	3.4.0
cloudpickle	3.1.2
google-crc32c	1.8.0
msgpack	1.2.2
anndata	0.13.3.post0
msgspec	0.21.1
fsspec	2026.7.0
MarkupSafe	3.0.3
kiwisolver	1.5.1
dask	2026.8.0
pydantic-settings	2.15.0
annotated-types	0.8.0
Jinja2	3.1.6
PyYAML	6.0.3
cycler	0.12.1
pydantic	2.13.5
pydantic_core	2.46.5
charset-normalizer	3.5.1
typing_extensions	4.16.0
sparse	0.19.2
h5py	3.16.0
session-info2	0.4.2
----	----
Python	3.13.13 (main, Apr  7 2026, 18:19:01) [Clang 21.0.0 (clang-2100.0.123.102)]
OS	macOS-26.6.2-arm64-arm-64bit-Mach-O
CPU	10/10 logical CPU cores, arm
GPU	No GPU found
Updated	2026-09-17 08:42

Start at _highly_variable_genes_batched and trace how the grouped DataFrame is reordered before adata._inplace_subset_var. Reproduce the mismatch with the minimal example, then extend test_subset_inplace_consistency to cover the default cutoff branch and verify that subset=True, subset=False, and returned frames preserve the same HVG selection and variable order.