aws / aws/sagemaker-python-sdk

SageMaker Processing Job doesn't support FastFile Input Mode

Đang mở
#4,711 3 bình luận 2 reaction 1 người được giao Được @pintaoz-aws nhận Xem trên GitHub
component: processing type: bug
Ngôn ngữ chính
Python
Star
2.3k
Fork
1.3k
Merge trung bình
1 ngày 22 giờ
Pull request đã merge (30 ngày)
35

Mô tả

**SageMaker Processing Job doesn't support FastFile Input Mode**
This might be the issue only for the step function, and python-sdk might provide the FastFile mode. However it doesn't make sense to me that one provides functionality and one doesn't. So, I post this issue.

According to this [document](https://sagemaker.readthedocs.io/en/stable/api/training/processing.html#sagemaker.processing.ProcessingInput), the ProcessingInput accepts FastFile mode. However If I try to create Processing Job using Step Function, error occurs.
```
"Error": "SageMaker.AmazonSageMakerException",
"Cause": "1 validation error detected: Value 'FastFile' at 'processingInputs.2.member.s3Input.s3InputMode' failed to satisfy constraint: Member must satisfy enum value set: [Pipe, File] (Service: AmazonSageMaker; Status Code: 400; Error Code: ValidationException; Request ID: 05f50214-59f0-4518-bd8e-36d800b078d0; Proxy: null)"
```
It only accepts Pipe, File.
This is related to this closed issue [#3962](https://github.com/aws/sagemaker-python-sdk/issues/3962)
And the document is updated by this PR[#4311](https://github.com/aws/sagemaker-python-sdk/pull/4311)

**Step Function Configuration**
![image](https://github.com/aws/sagemaker-python-sdk/assets/30307587/e622bfb4-f18a-44f4-93c6-aacec64ce394)

```json
{
"ProcessingResources": {
"ClusterConfig": {
"InstanceCount": 1,
"InstanceType.$": "$.inferenceOpt.instanceType",
"VolumeSizeInGB": 100
}
},
"AppSpecification": {
"ImageUri.$": "$.inferenceOpt.imageUri",
"ContainerEntrypoint": [
"python3",
"/opt/ml/code/inference.py"
]
},
"Environment": {
"FINAL_KERNEL_SIZE.$": "$.hyperparameters.FINAL_KERNEL_SIZE",
"MODEL_ARCH.$": "$.hyperparameters.MODEL_ARCH",
"LAST_LEVEL.$": "$.hyperparameters.LAST_LEVEL",
"IMG_SIZE.$": "$.hyperparameters.IMG_SIZE",
"NUM_CLASSES.$": "$.hyperparameters.NUM_CLASSES",
"CLASS_DICT.$": "$.hyperparameters.CLASS_DICT",
"PREDICT_DATASET_ID.$": "$.hyperparameters.DATASET_ID",
"S3_PREFIX.$": "$.hyperparameters.SERVICE_NAME",
"PREDICT_ID.$": "$.hyperparameters.PREDICT_ID",
"MANIFEST_FILE.$": "$.inferenceOpt.manifestFile",
"QUEUE_URL": "https://sqs.ap-northeast-2.amazonaws.com/405240163678/Prod-Proto-CvOpsApplication-ProtoInferenceMetricQueueProtoInference-SgfJZK5ae1TV",
"REGION": "ap-northeast-2"
},
"NetworkConfig": {
"EnableNetworkIsolation": false,
"EnableInterContainerTrafficEncryption": true,
"VpcConfig": {
"SecurityGroupIds": [
"sg-0aee4467bff25504b"
],
"Subnets": [
"subnet-06dc8a4e813910a07",
"subnet-0b023fbbb320cf8d8",
"subnet-0e3b92aad033b1993"
]
}
},
"ProcessingInputs": [
{
"InputName": "model",
"S3Input": {
"S3Uri.$": "$.inferenceOpt.modelArtifact",
"LocalPath": "/opt/ml/processing/model",
"S3DataType": "S3Prefix",
"S3InputMode": "File",
"S3DataDistributionType": "FullyReplicated",
"S3CompressionType": "None"
}
},
{
"InputName": "dataset",
"S3Input": {
"S3Uri.$": "$.inferenceOpt.datastoreName",
"LocalPath": "/opt/ml/processing/manifest",
"S3DataType": "S3Prefix",
"S3InputMode": "FastFile",
"S3DataDistributionType": "FullyReplicated",
"S3CompressionType": "None"
}
},
{
"InputName": "manifest",
"S3Input": {
"S3Uri.$": "$.inferenceOpt.manifestFile",
"LocalPath": "/opt/ml/processing/config",
"S3DataType": "S3Prefix",
"S3InputMode": "File",
"S3DataDistributionType": "FullyReplicated",
"S3CompressionType": "None"
}
}
],
"RoleArn": "arn:aws:iam::405240163678:role/Prod-Proto-CvOpsApplicati-ProtoInferenceWorkflowPro-Kh3pe5GGN20F",
"ProcessingJobName.$": "States.Format('cvops-inference-{}', $$.Execution.Name)",
"Tags": [
{
"Key": "CVOps",
"Value": "Proto-CvopsInferenceTask"
}
]
}
```

**Expected behavior**
Accepting the FastFile mode, or updating the document

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.