aws / aws/sagemaker-python-sdk

Environment variables limited to 256 characters

Đang mở
#2,932 4 bình luận 12 reaction 1 người được giao Được @rsareddy0329 nhận Xem trên GitHub
component: pysdk-team type: feature request
Ngôn ngữ chính
Python
Star
2.3k
Fork
1.3k
Merge trung bình
1 ngày 22 giờ
Pull request đã merge (30 ngày)
35

Mô tả

**Describe the bug**
Currently, the value length constraints for environment variables are quite limited in `CreateProcessingJob` and `CreateTrainingJob`.

* [CreateProcessingJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateProcessingJob.html#sagemaker-CreateProcessingJob-request-Environment): 256
* [CreateTrainingJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTrainingJob.html#sagemaker-CreateTrainingJob-request-Environment): 512

This causes issues when passing urls or other long strings via the environment parameter, as they often exceed the limit. I am not aware of an alternative to pass items that shouldn't be written to files to a job at runtime. Storing some info on s3 and treating it as an input cannot be used as a workaround, as some variables should not be shared with everyone who has access to the s3 bucket nor stored long term. It's also inconvenient to write a notebook/user's env vars to s3, read them in, set them as env vars etc.

It looks like it would be possible to have an api call accept keys that are over 256 characters in length. Example https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTrainingJob.html#sagemaker-CreateTrainingJob-request-HyperParameters accepts 2500 characters.

**To reproduce**
Pass a string over the char limit similar to below.

```python
estimator = sagemaker.estimator.Estimator(
image,
role,
environment={"LONG_STRING": "".join(("0" for _ in range(513)))},
instance_count=1,
instance_type=instance_type,
)

script_processor = sagemaker.processing.ScriptProcessor(
command=["my", "command"],
env={"LONG_STRING": "".join(("0" for _ in range(257)))},
image_uri=image,
role=role,
instance_count=1,
instance_type=instance_type,
)

```

**Expected behavior**
Ability to pass longer environment variables to jobs.

**Screenshots or logs**

```console
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the CreateProcessingJob operation: 1 validation error detected: Value '' at 'environment' failed to satisfy constraint: Map value must satisfy constraint: [Member must have length less than or equal to 256, Member must have length greater than or equal to 0, Member must satisfy regular expression pattern: [\S\s]*]
```

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: 2.52.0
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: N/A
- **Framework version**: N/A
- **Python version**: 3.9
- **CPU or GPU**: CPU
- **Custom Docker image (Y/N)**: Both

**Additional context**
I suspect others may be facing the same issue as there is mention of passing custom urls for pip configurations.

https://github.com/aws/sagemaker-python-sdk/issues/2207#issuecomment-810096332

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.