XGBoostError: input stream corrupted when unpickling a Linux-trained Booster on Windows (xgboost 3.3.0)
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
## Summary
A scikit-learn `Pipeline` wrapping an `XGBClassifier`, pickled with `joblib` on **Linux** with xgboost **3.3.0**, fails to unpickle on **Windows** with the identical xgboost **3.3.0** version:
```
xgboost._c_api.XGBoostError: input stream corrupted
```
The failure happens inside `Booster.__setstate__` → `XGBoosterUnserializeFromBuffer`. The same pickle file loads without any issue on Linux (native and under WSL2). This is not a version mismatch, not a corrupted upload, and not a locale/decimal-separator issue (all three were tested and ruled out — details below). It looks like a regression specific to the Windows wheel's native deserializer.
## Environment
**Producer (save side) — native Linux:**
- OS: `Linux-7.0.0-28-generic-x86_64-with-glibc2.43` (Ubuntu)
- Python: CPython 3.12.13
- xgboost: `3.3.0`, wheel tag `py3-none-manylinux_2_28_x86_64`
- scikit-learn: `1.9.0`
- numpy: `2.3.5`
**Producer (save side) — WSL2 Linux, used to build the minimal repro below:**
- OS: `Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.39`
- Python: 3.12.3 (GCC 13.3.0)
- xgboost: `3.3.0`, wheel tag `py3-none-manylinux_2_28_x86_64`
- scikit-learn: `1.9.0`
**Consumer (load side) — Windows, where it fails:**
- OS: `Windows-10-10.0.19045-SP0`
- Python: 3.12.13 (MSC v.1944 64 bit AMD64)
- xgboost: `3.3.0`, wheel tag `py3-none-win_amd64`
- scikit-learn: `1.9.0`
## Minimal reproduction
**On Linux** (native or WSL2):
```python
import joblib
from sklearn.datasets import make_classification
from sklearn.pipeline import Pipeline
from xgboost import XGBClassifier
X, y = make_classification(n_samples=200, n_features=5, random_state=42)
pipe = Pipeline([("clf", XGBClassifier(n_estimators=10, max_depth=3, random_state=42))])
pipe.fit(X, y)
joblib.dump(pipe, "minimal_model.pkl")
```
This produces a 21,597-byte `.pkl` file. Copy it to a Windows machine (same xgboost version, verified `3.3.0` on both sides) and run:
```python
import joblib
pipe = joblib.load("minimal_model.pkl")
```
**Expected:** loads successfully, as it does on Linux.
**Actual:** raises immediately.
## Full traceback (Windows)
```
Traceback (most recent call last):
File "", line 1, in
File "...\joblib\numpy_pickle.py", line 749, in load
obj = _unpickle(...)
File "...\joblib\numpy_pickle.py", line 626, in _unpickle
obj = unpickler.load()
File "...\Lib\pickle.py", line 1256, in load
dispatch[key[0]](self)
File "...\joblib\numpy_pickle.py", line 446, in load_build
Unpickler.load_build(self)
File "...\Lib\pickle.py", line 1760, in load_build
setstate(state)
File "...\xgboost\core.py", line 1885, in __setstate__
_check_call(_LIB.XGBoosterUnserializeFromBuffer(handle, ptr, length))
File "...\xgboost\_c_api.py", line 190, in _check_call
raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost._c_api.XGBoostError: input stream corrupted
```
i.e. `Booster.__getstate__` (Linux side) serializes via `XGBoosterSerializeToBuffer` into a `bytearray`, joblib pickles that `bytearray` as-is, and `Booster.__setstate__` (Windows side) passes those exact bytes into `XGBoosterUnserializeFromBuffer`, which rejects them.
## What we ruled out
1. **xgboost version mismatch** — confirmed identical `xgboost==3.3.0` on both the producing (Linux) and consuming (Windows) environments, both from the requirements/lockfile actually used at save/load time.
2. **Corrupted/partial file transfer** — the buffer inside the pickle is a well-formed, complete UBJSON stream: it begins with a normal object header (`{L\x00...\x06Config{...`) and ends with a correctly closed top-level object including a `"version"` key whose value correctly encodes `[3, 3, 0]`. Extracting the raw bytes (by monkeypatching `Booster.__setstate__` to capture `state["handle"]` before it's passed to the native call) and inspecting them shows no truncation or obviously malformed byte sequences. A joblib/pickle-level integrity issue would also have failed earlier, during unpickling of the `bytearray` itself, not inside `Booster.__setstate__`'s native call.
3. **Locale / decimal-separator parsing** — the Windows machine's OS locale is `pt_BR` (comma decimal separator). We forced the Python process to `locale.setlocale(locale.LC_ALL, "C")` before loading and the failure was identical, ruling out a `strtod`/`printf`-style locale-dependent number-parsing issue reachable from Python.
4. **Original real-world case wasn't a fluke** — the same failure was first observed with a production churn-prediction model (a `Pipeline` with feature preprocessing + `XGBClassifier`, ~4.2MB `.pkl`) trained on Linux and shared via a Weights & Biases model artifact; it fails on Windows and loads fine on both native Linux and WSL2 with the same xgboost version. The 21KB synthetic reproducer above confirms it isn't specific to that model's size, feature set, or hyperparameters.
## Notes
- A model trained **and** loaded on the same Windows machine (no cross-platform pickle) round-trips fine — the bug only manifests when the pickle crosses from a Linux-built xgboost wheel to the Windows-built one.
- Not yet tested: Windows → Linux direction (pickle produced on Windows, loaded on Linux). Worth checking whether this is one-directional or symmetric.
- Happy to attach the 21,597-byte [minimal_model.zip](https://github.com/user-attachments/files/31048518/minimal_model.zip) reproducer file directly to the issue, or provide the raw extracted UBJSON buffer bytes, if useful for debugging.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the minimal Linux-to-Windows reproduction and inspect Booster.__setstate__ in xgboost/core.py, which calls XGBoosterUnserializeFromBuffer through xgboost/_c_api.py. Compare the native serialization and deserialization paths, including the Windows and Linux builds, then verify that the supplied 21,597-byte model loads cross-platform and that the relevant tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- machine-learning, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100