aws / aws/sagemaker-python-sdk

Environment variables limited to 256 characters

オープン
#2,932 コメント 4 件 リアクション 12 件 担当者 1 名 @rsareddy0329 が担当を希望しています GitHub で見る
component: pysdk-team type: feature request
主要言語
Python
スター
2.3k
フォーク
1.3k
平均マージ
1日 22時間
マージ済み PR(30日)
35

説明

**Describe the bug**
Currently, the value length constraints for environment variables are quite limited in `CreateProcessingJob` and `CreateTrainingJob`.

* [CreateProcessingJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateProcessingJob.html#sagemaker-CreateProcessingJob-request-Environment): 256
* [CreateTrainingJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTrainingJob.html#sagemaker-CreateTrainingJob-request-Environment): 512

This causes issues when passing urls or other long strings via the environment parameter, as they often exceed the limit. I am not aware of an alternative to pass items that shouldn't be written to files to a job at runtime. Storing some info on s3 and treating it as an input cannot be used as a workaround, as some variables should not be shared with everyone who has access to the s3 bucket nor stored long term. It's also inconvenient to write a notebook/user's env vars to s3, read them in, set them as env vars etc.

It looks like it would be possible to have an api call accept keys that are over 256 characters in length. Example https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTrainingJob.html#sagemaker-CreateTrainingJob-request-HyperParameters accepts 2500 characters.

**To reproduce**
Pass a string over the char limit similar to below.

```python
estimator = sagemaker.estimator.Estimator(
image,
role,
environment={"LONG_STRING": "".join(("0" for _ in range(513)))},
instance_count=1,
instance_type=instance_type,
)

script_processor = sagemaker.processing.ScriptProcessor(
command=["my", "command"],
env={"LONG_STRING": "".join(("0" for _ in range(257)))},
image_uri=image,
role=role,
instance_count=1,
instance_type=instance_type,
)

```

**Expected behavior**
Ability to pass longer environment variables to jobs.

**Screenshots or logs**

```console
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the CreateProcessingJob operation: 1 validation error detected: Value '' at 'environment' failed to satisfy constraint: Map value must satisfy constraint: [Member must have length less than or equal to 256, Member must have length greater than or equal to 0, Member must satisfy regular expression pattern: [\S\s]*]
```

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: 2.52.0
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: N/A
- **Framework version**: N/A
- **Python version**: 3.9
- **CPU or GPU**: CPU
- **Custom Docker image (Y/N)**: Both

**Additional context**
I suspect others may be facing the same issue as there is mention of passing custom urls for pip configurations.

https://github.com/aws/sagemaker-python-sdk/issues/2207#issuecomment-810096332

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。