aws / aws/sagemaker-python-sdk
Environment variables limited to 256 characters
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 1.3k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 35
Description
**Describe the bug**
Currently, the value length constraints for environment variables are quite limited in `CreateProcessingJob` and `CreateTrainingJob`.
* [CreateProcessingJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateProcessingJob.html#sagemaker-CreateProcessingJob-request-Environment): 256
* [CreateTrainingJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTrainingJob.html#sagemaker-CreateTrainingJob-request-Environment): 512
This causes issues when passing urls or other long strings via the environment parameter, as they often exceed the limit. I am not aware of an alternative to pass items that shouldn't be written to files to a job at runtime. Storing some info on s3 and treating it as an input cannot be used as a workaround, as some variables should not be shared with everyone who has access to the s3 bucket nor stored long term. It's also inconvenient to write a notebook/user's env vars to s3, read them in, set them as env vars etc.
It looks like it would be possible to have an api call accept keys that are over 256 characters in length. Example https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTrainingJob.html#sagemaker-CreateTrainingJob-request-HyperParameters accepts 2500 characters.
**To reproduce**
Pass a string over the char limit similar to below.
```python
estimator = sagemaker.estimator.Estimator(
image,
role,
environment={"LONG_STRING": "".join(("0" for _ in range(513)))},
instance_count=1,
instance_type=instance_type,
)
script_processor = sagemaker.processing.ScriptProcessor(
command=["my", "command"],
env={"LONG_STRING": "".join(("0" for _ in range(257)))},
image_uri=image,
role=role,
instance_count=1,
instance_type=instance_type,
)
```
**Expected behavior**
Ability to pass longer environment variables to jobs.
**Screenshots or logs**
```console
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the CreateProcessingJob operation: 1 validation error detected: Value '' at 'environment' failed to satisfy constraint: Map value must satisfy constraint: [Member must have length less than or equal to 256, Member must have length greater than or equal to 0, Member must satisfy regular expression pattern: [\S\s]*]
```
**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: 2.52.0
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: N/A
- **Framework version**: N/A
- **Python version**: 3.9
- **CPU or GPU**: CPU
- **Custom Docker image (Y/N)**: Both
**Additional context**
I suspect others may be facing the same issue as there is mention of passing custom urls for pip configurations.
https://github.com/aws/sagemaker-python-sdk/issues/2207#issuecomment-810096332
Contributor guide
Assessment
This issue has not been assessed yet.