Serverless Endpoint Can't Run Due to Insufficient Space

**Describe the bug**
I am trying to run a serverless endpoint but the endpoint always fails to get created while trying to install dependencies. I understand that serverless endpoints do not have much space but I provisioned the full 6GB amount and it hasn't even gotten to downloading the model.

**To reproduce**
Create a sagemaker serverless endpoint withe the following configuration:

IMAGE:

763104351884.dkr.ecr.eu-west-2.amazonaws.com/pytorch-inference:2.2.0-cpu-py310-ubuntu20.04-sagemaker

REQUIRMENTS:

torchaudio==2.2.2
sox==1.5.0
huggingface_hub>=0.8.0
hyperpyyaml>=0.0.1
joblib>=0.14.1
numpy>=1.17.0
packaging
pandas>=1.0.1
pre-commit>=2.3.0
pygtrie>=2.1,<3.0
scipy>=1.4.1,<1.13.0
sentencepiece>=0.1.91
SoundFile; sys_platform == 'win32'
torch>=1.9.0,<=2.2.2
tqdm>=4.42.0
transformers>=4.30.0
speechbrain==1.0.0

Alternatively you could reduce this to the following but the others will be installed as dependencies anyway:

torchaudio==2.2.2
sox==1.5.0
speechbrain==1.0.0

MEMORY:

6GB

**Expected behavior**
Serverless endpoint is created

**Screenshots or logs**

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: Used AWS Console
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: Pytorch, Speechbrain (speechbrain/spkrec-ecapa-voxceleb)
- **Framework version**: Speechbrain (1.0.0)
- **Python version**: 3.10
- **CPU or GPU**: CPU
- **Custom Docker image (Y/N)**: N

**Additional context**
On my local machine a virtual environment with the packages outlined in the requirements.txt file takes 842MB