[BUG] ValueError: Input X contains infinity or a value too large for dtype('float32')

**Bug Report Checklist**

- [x] I provided code that demonstrates a minimal reproducible example.
- [ ] I confirmed bug exists on the latest mainline of AutoGluon via source install.
- [x] I confirmed bug exists on the latest stable version of AutoGluon.

**Describe the bug**

Error as per title when doing multimodal few shot classification.
Error is present when following the tutorial: https://auto.gluon.ai/stable/tutorials/multimodal/advanced_topics/few_shot_learning.html

**Expected behavior**

The process would not error.

**To Reproduce**

Error is present when replicating example in tutorial: https://auto.gluon.ai/stable/tutorials/multimodal/advanced_topics/few_shot_learning.html

**Screenshots / Logs**

```
❌ ValueError: Input X contains infinity or a value too large for dtype('float32').
Traceback (most recent call last):
File "", line 23, in
File "/opt/homebrew/Caskroom/miniforge/base/envs/py311_knime52_autogluon/lib/python3.11/site-packages/autogluon/multimodal/predictor.py", line 509, in fit
self._learner.fit(
File "/opt/homebrew/Caskroom/miniforge/base/envs/py311_knime52_autogluon/lib/python3.11/site-packages/autogluon/multimodal/learners/base.py", line 654, in fit
fit_returns = self.execute_fit()
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/py311_knime52_autogluon/lib/python3.11/site-packages/autogluon/multimodal/learners/base.py", line 566, in execute_fit
attributes = self.fit_per_run(**self._fit_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/py311_knime52_autogluon/lib/python3.11/site-packages/autogluon/multimodal/learners/few_shot_svm.py", line 359, in fit_per_run
svm.fit(features, labels)
File "/opt/homebrew/Caskroom/miniforge/base/envs/py311_knime52_autogluon/lib/python3.11/site-packages/sklearn/base.py", line 1152, in wrapper
return fit_method(estimator, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/py311_knime52_autogluon/lib/python3.11/site-packages/sklearn/pipeline.py", line 423, in fit
Xt = self._fit(X, y, **fit_params_steps)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/py311_knime52_autogluon/lib/python3.11/site-packages/sklearn/pipeline.py", line 377, in _fit
X, fitted_transformer = fit_transform_one_cached(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/py311_knime52_autogluon/lib/python3.11/site-packages/joblib/memory.py", line 312, in __call__
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/py311_knime52_autogluon/lib/python3.11/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
res = transformer.fit_transform(X, y, **fit_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/py311_knime52_autogluon/lib/python3.11/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
data_to_wrap = f(self, X, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/py311_knime52_autogluon/lib/python3.11/site-packages/sklearn/base.py", line 919, in fit_transform
return self.fit(X, y, **fit_params).transform(X)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/py311_knime52_autogluon/lib/python3.11/site-packages/sklearn/preprocessing/_data.py", line 839, in fit
return self.partial_fit(X, y, sample_weight)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/py311_knime52_autogluon/lib/python3.11/site-packages/sklearn/base.py", line 1152, in wrapper
return fit_method(estimator, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/py311_knime52_autogluon/lib/python3.11/site-packages/sklearn/preprocessing/_data.py", line 875, in partial_fit
X = self._validate_data(
^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/py311_knime52_autogluon/lib/python3.11/site-packages/sklearn/base.py", line 605, in _validate_data
out = check_array(X, input_name="X", **check_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/py311_knime52_autogluon/lib/python3.11/site-packages/sklearn/utils/validation.py", line 957, in check_array
_assert_all_finite(
File "/opt/homebrew/Caskroom/miniforge/base/envs/py311_knime52_autogluon/lib/python3.11/site-packages/sklearn/utils/validation.py", line 122, in _assert_all_finite
_assert_all_finite_element_wise(
File "/opt/homebrew/Caskroom/miniforge/base/envs/py311_knime52_autogluon/lib/python3.11/site-packages/sklearn/utils/validation.py", line 171, in _assert_all_finite_element_wise
raise ValueError(msg_err)
```

**Installed Versions**

INSTALLED VERSIONS
------------------
date : 2024-08-13
time : 12:12:41.286993
python : 3.11.9.final.0
OS : Darwin
OS-release : 23.5.0
Version : Darwin Kernel Version 23.5.0: Wed May 1 20:12:58 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6000
machine : arm64
processor : arm
num_cores : 10
cpu_ram_mb : 32768.0
cuda version : None
num_gpus : 0
gpu_ram_mb : []
avail_disk_size_mb : 284332

accelerate : 0.21.0
autogluon : 1.1.1
autogluon.common : 1.1.1
autogluon.core : 1.1.1
autogluon.features : 1.1.1
autogluon.multimodal : 1.1.1
autogluon.tabular : 1.1.1
autogluon.timeseries : 1.1.1
boto3 : 1.34.158
catboost : 1.2.5
defusedxml : 0.7.1
evaluate : 0.4.1
fastai : 2.7.16
gluonts : 0.14.3
hyperopt : 0.2.7
imodels : None
jinja2 : 3.1.4
joblib : 1.4.2
jsonschema : 4.21.1
lightgbm : 4.3.0
lightning : 2.3.3
matplotlib : 3.8.2
mlforecast : 0.10.0
networkx : 3.3
nlpaug : 1.1.11
nltk : 3.8.1
nptyping : 2.4.1
numpy : 1.26.4
nvidia-ml-py3 : None
omegaconf : 2.3.0
onnxruntime-gpu : None
openmim : 0.3.7
optimum : None
optimum-intel : None
orjson : 3.10.7
pandas : 2.0.3
pdf2image : 1.17.0
Pillow : 10.1.0
psutil : 5.9.8
pytesseract : 0.3.10
pytorch-lightning : 2.3.3
pytorch-metric-learning: 2.3.0
ray : 2.31.0
requests : 2.31.0
scikit-image : 0.20.0
scikit-learn : 1.3.2
scikit-learn-intelex : None
scipy : 1.11.4
seqeval : 1.2.2
setuptools : 72.1.0
skl2onnx : None
statsforecast : 1.4.0
tabpfn : None
tensorboard : 2.17.0
text-unidecode : 1.3
timm : 0.9.16
torch : 2.2.2
torchmetrics : 1.2.1
torchvision : 0.17.2
tqdm : 4.66.5
transformers : 4.38.2
utilsforecast : 0.0.10
vowpalwabbit : None
xgboost : 2.0.3

Start by reproducing the few-shot classification example from the linked multimodal tutorial with the reported AutoGluon 1.1.1 environment. Inspect autogluon/multimodal/learners/few_shot_svm.py around line 359 and the scikit-learn pipeline shown in the traceback. Done means the tutorial flow completes without the reported infinity or float32 ValueError.

Written by the indexing model from the issue text.