Multiple runs on same data with early_stopping not working in GPU training with dask_cudf

Hi,

currently I am working on a multi-gpu workflow using dask_cudf together with xgboost 1.7.1 and early stopping.

The basic workflow works fine. However, if I want to iterate multiple times over the same data, e.g. with different parameters, it appears that the early stopping is only working for the first iteration. In subsequent iterations training is not starting and it only prints the validation loss for node 0 and then is finished with training without any error:

The basic workflow looks like this:
```python
cluster = LocalCUDACluster(n_workers=4)
client = Client(cluster)

df = dask_cudf.read_parquet("path_to_parquet")

X_train = df[df["CalDay"] < validation_start_date][features]
y_train = df[df["CalDay"] < validation_start_date][label]

X_val = df[df["CalDay"] >= validation_start_date][features]
y_val = df[df["CalDay"] >= validation_start_date][label]

def fit_xgboost(params, X_train, y_train, X_val, y_val):
es = xgb.callback.EarlyStopping(rounds=early_stopping_rounds, save_best=False)

Xy = dxgb.DaskQuantileDMatrix(client, X_train, y_train)
Xy_valid = dxgb.DaskDMatrix(client, X_val, y_val)

booster = xgb.dask.train(
client,
params,
Xy,
evals=[(Xy_valid, "Valid")],
num_boost_round=1000,
callbacks=[es],
verbose_eval=True
)
return booster

for params in params_list:
fit_xgboost(params, X_train, y_train, X_val, y_val)
```

Thanks for any help that you can provide!

Reproduce the multi-GPU example around fit_xgboost, DaskQuantileDMatrix, DaskDMatrix, and xgb.dask.train, running the same data through params_list with the EarlyStopping callback. Compare the first and subsequent iterations, including the distributed validation output. Done means repeated training starts normally and early stopping behaves consistently for each parameter set.

Written by the indexing model from the issue text.