Construction of arrays with `object` dtype very slow when Pandas `future.infer_string` is enabled
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What happened?
Constructing a Variable (or Dataset/DataArray) from an object-dtype numpy array of strings takes ~0.1ms per 10M elements under pandas' default settings, but ~500–620ms with pd.options.future.infer_string = True — the setting that becomes the default in pandas 3.0.
Why:
as_compatible_data routes every object array through _possibly_convert_objects,
which round-trips the data through pd.Series to convert datetime/timedelta objects to datetime64/timedelta64.
Under legacy (Pandas 2.x) inference, that probe was effectively free — pandas' datetime scan bails on the first non-datetime element. Under string (Pandas 3.x) inference, the same constructor's contract now includes instantiating the string dtype object, so pandas performs a full scan and materializes an Arrow string array of the entire input — which _possibly_convert_objects immediately converts back to numpy and discards:
as_compatible_data
└─ _possibly_convert_objects
└─ pd.Series(values) # probe only
└─ maybe_infer_to_datetimelike
└─ ArrowStringArrayNumpySemantics._from_sequence # full Arrow build
└─ np.asarray(series) # Arrow array discarded
Minimal Complete Verifiable Example
import time
import numpy as np
import pandas as pd
import xarray as xr
rng = np.random.default_rng(0)
labels = np.array([f"label_{i:04d}" for i in range(500)], dtype=object)
arr = labels[rng.integers(0, len(labels), 10_000_000)]
for infer_string in (False, True):
pd.options.future.infer_string = infer_string
best = float("inf")
for _ in range(5):
t0 = time.perf_counter()
xr.Variable(("x",), arr)
best = min(best, time.perf_counter() - t0)
print(f"infer_string={infer_string}: {best * 1000:.1f}ms")
Output:
infer_string=False: 0.1ms
infer_string=True: 606.3ms
MVCE confirmation
- Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- Complete example — the example is self-contained, including all data and the text of any traceback.
- Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
- New issue — a search of GitHub Issues suggests this is not a duplicate.
- Recent environment — the issue occurs with the latest version of xarray and its dependencies.
Environment
INSTALLED VERSIONS
commit: None
python: 3.14.5 (main, May 10 2026, 19:28:16) [Clang 22.1.3 ]
python-bits: 64
OS: Linux
OS-release: 6.17.0-1019-aws
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: C.UTF-8
LOCALE: ('C', 'UTF-8')
libhdf5: None
libnetcdf: None
xarray: 2026.7.0
pandas: 2.3.3
numpy: 2.4.4
scipy: 1.17.1
netCDF4: None
pydap: None
h5netcdf: None
h5py: None
zarr: 2.18.7
cftime: None
nc_time_axis: None
iris: None
bottleneck: 1.6.0
dask: 2025.12.0
distributed: None
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2026.4.0
cupy: None
pint: None
sparse: 0.18.0
flox: None
numpy_groupies: None
setuptools: 80.9.0
pip: None
conda: None
pytest: 9.1.0
mypy: None
IPython: 9.12.0
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in xarray/core/variable.py at as_compatible_data and _possibly_convert_objects, especially the linked lines around the pd.Series probe. Reproduce the slowdown with the provided MVCE under both pandas inference settings, then verify that the datetime/timedelta conversion behavior remains correct without materializing a discarded Arrow string array; done means the object-string construction no longer incurs the reported slowdown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100