allowed_model_families is not enforced

I want evalml to only create XGBOOST models in the allotted time, but both baseline and Random Forest models are also generated.
My hyperparameters and my setup are:

#### Code Sample, a copy-pastable example to reproduce your bug.

```python
eval_automl = automl.AutoMLSearch(
X_train=train_data,
y_train=train_target,
problem_type="binary",
max_time=60,
allowed_model_families=["xgboost"],
ensembling=False,
n_jobs=1,
verbose=True,
)
```
I understand the baseline could be mandatory to ensure a result, but why is random forests also included?