Fail to launch Processing s3_data_distribution_type='ShardedByS3Key'

I'm currently trying to run a parallel processing job on sagemaker and the data needs to be processed across multiple instances and then combined into one output. When attempting to launch the processing job it just fails to launch and after trial and error I realized it falls down to adding the sharded files to the inputs argument of the ProcessingStep.

Below is a sample of a subset of the data that I divided into roughly equal chunks:

Defining the parameter integers and string as shown below:

processing_instance_count = ParameterInteger(
name="ProcessingInstanceCount", default_value=2
)
processing_instance_type = ParameterString(
name="ProcessingInstanceType", default_value="ml.m5.2xlarge"
)

date_window_start = ParameterString(
name="DateWindowStart", default_value='2000-01-01'
)
date_window_end = ParameterString(
name="DateWindowEnd", default_value='3000-01-01'
)

test_leases = ParameterString(
name="TestLeasesUri",
default_value='s3://bucket123/foo/bar/output/test/test.h5'
)

all_leases = ParameterString(
name="AllLeasesUri",
default_value='s3://mybucket/username/foo/data/all_lease_docs_fraction_chunks/'
)

labels_dir = ParameterString(
name='LabelsDir',
default_value='s3://bucket123/foo/bar/labels.json'
)

`

Below is the processing definition:

```
processor = ScriptProcessor(
image_uri=preprocess_image_uri,
instance_type=processing_instance_type,
instance_count=processing_instance_count,
base_job_name="preprocessSeqLblr",
sagemaker_session=sagemaker_session,
role=role,
command=["python"],
network_config=network_config,
max_runtime_in_seconds=36_000,
volume_size_in_gb = 250 #assign larger storage
)
```

```
step_process = ProcessingStep(
name="Preprocess",
processor=processor,
inputs=[
ProcessingInput(
source="s3://bucket123/SeqLblr_labels.txt",
input_name = 'SeqLblr_Labels',
s3_data_distribution_type = 'FullyReplicated',
destination="/opt/ml/processing/labels",
),
ProcessingInput(
source="s3://sagemaker-us-west-2-bucket123/foo/bar/output/test/test.h5",
input_name = 'TestLeases',
s3_data_distribution_type = 'FullyReplicated',
destination="/opt/ml/processing/test_leases",
),
ProcessingInput(
source="s3://mybucket/username/foo/data/all_lease_docs_fraction_chunks/",
input_name='AllLeasesUri',
s3_data_distribution_type='ShardedByS3Key',
s3_data_type='S3Prefix',
destination="opt/ml/processing/all_leases",
)
],
outputs=[
ProcessingOutput(output_name="output", source="/opt/ml/processing/output")
],
code=os.path.join(BASE_DIR, "preprocess.py"),
job_arguments=[
"--labels-dir", labels_dir,
"--test-leases", test_leases,
"--all-leases", all_leases,
'--date-window-start', date_window_start,
'--date-window-end', date_window_end
],
cache_config=cache_config,
)

```

`

I am invoking the pipelien via the following method inside a sagemaker studio notebook.
```

from pipelines.SeqLblr.pipeline import get_pipeline
role = 'myrole123'

pipeline = get_pipeline(
region=region,
role=role,
default_bucket=default_bucket,
model_package_group_name=model_package_group_name,
pipeline_name=pipeline_name,
)

pipeline.upsert(role_arn=role)

execution = pipeline.start(
parameters=dict(
DateWindowStart='2000-01-01',
DateWindowEnd='3000-01-01',
AllLeasesUri='s3://mybucket/username/foo/data/all_lease_docs_fraction_chunks/',
LabelsDir = 's3://bucket123/foo/bar/labels.json',
TestLeasesUri='s3://bucket123/foo/bar/output/test/test.h5'

)
)

```

Note that when I switch out the AllLeasesURI parameter with the entire dataset combined and keep the instance_count=2, it launched 2 fully replicated instances so I don't think it's a quota issue.