Serverless Endpoint Can't Run Due to Insufficient Space

**Describe the bug**
I am trying to run a serverless endpoint but the endpoint always fails to get created while trying to install dependencies. I understand that serverless endpoints do not have much space but I provisioned the full 6GB amount and it hasn't even gotten to downloading the model.

**To reproduce**
Create a sagemaker serverless endpoint withe the following configuration:

IMAGE:

763104351884.dkr.ecr.eu-west-2.amazonaws.com/pytorch-inference:2.2.0-cpu-py310-ubuntu20.04-sagemaker

REQUIRMENTS:

torchaudio==2.2.2
sox==1.5.0
huggingface_hub>=0.8.0
hyperpyyaml>=0.0.1
joblib>=0.14.1
numpy>=1.17.0
packaging
pandas>=1.0.1
pre-commit>=2.3.0
pygtrie>=2.1,<3.0
scipy>=1.4.1,<1.13.0
sentencepiece>=0.1.91
SoundFile; sys_platform == 'win32'
torch>=1.9.0,<=2.2.2
tqdm>=4.42.0
transformers>=4.30.0
speechbrain==1.0.0

Alternatively you could reduce this to the following but the others will be installed as dependencies anyway:

torchaudio==2.2.2
sox==1.5.0
speechbrain==1.0.0

MEMORY:

6GB

**Expected behavior**
Serverless endpoint is created

**Screenshots or logs**

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: Used AWS Console
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: Pytorch, Speechbrain (speechbrain/spkrec-ecapa-voxceleb)
- **Framework version**: Speechbrain (1.0.0)
- **Python version**: 3.10
- **CPU or GPU**: CPU
- **Custom Docker image (Y/N)**: N

**Additional context**
On my local machine a virtual environment with the packages outlined in the requirements.txt file takes 842MB

Le rapport ne mentionne aucun fichier, test ni version du SDK de SageMaker Python SDK ; il a été reproduit via l'AWS Console avec une image PyTorch 2.2.0 et les dépendances indiquées. Commencez par reproduire la création du endpoint serverless et examiner les journaux d'erreur de l'installation des dépendances. C'est terminé lorsque l'endpoint est créé avec succès dans la limite d'allocation de 6GB, ou lorsque la limitation est documentée.

Rédigé par le modèle d'indexation à partir du texte de l'issue.