aws / aws/sagemaker-python-sdk
Increase Length Limit for Arguments to Steps
- Langage dominant
- Python
- Étoiles
- 2.3k
- Forks
- 1.3k
- Merge moyen
- 1 j 22 h
- PR mergées (30 j)
- 35
Description
**Describe the feature you'd like**
Consider raising the character limit on arguments passed to scripts a little. Right now, each parameter passed to a step through `job_arguments` must be no more than 256 characters. This is an oppressive limit. Consider:
```
arguments = [
"--s3_data_path",
s3_path,
"--model_description",
# This, whether it's Parameter or str, cannot be > 256 chars
model_description,
]
preprocessing_step = ProcessingStep(
name=...,
processor=...,
code=...,
job_arguments=arguments,
)
```
As a result, when building pipelines, you end up having to upload even something simple like a description string to a file on S3. What's worse, is that if `model_description` in the example above is a `Parameter`, a `Parameter` can legally be up to 1024 characters. So not even every Parameter can be passed to a Step! Whether the above code will break or not, if `model_description` is a `Parameter`, depends on the Parameter's runtime value. Surely this is a very sad state of affairs.
Even just fairly modest increases to the size limits would greatly improve quality-of-life. At the very least, the size limit should match the size limit of ParameterString, at 1024 characters, so that any `Parameter` can be passed as a step argument. Ideally, for both ParameterString and the argument length, the limit would be increased further to a number more like 10k. That would allow any small content that's reasonable to have as a Parameter (e.g. small text blob) to be a Parameter while things like data files, etc. would still have to be uploaded to S3.
I am fairly confident that this could be done without hitting command length limitations on any modern shell. If the issue is that there may be up to 100 parameters, impose an _overall_ length limit on the shell command so that each individual parameter does not need to be so constrained.
**How would this feature be used? Please describe.**
Users would more easily pass more data directly, without having to put things in S3 files all the time, greatly increasing the usability of Sagemaker pipelines.
**Describe alternatives you've considered**
The workaround is that any content even potentially exceeding 256 characters must go in an S3 file.
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Commencez par localiser la validation de `job_arguments` et la gestion de la longueur de `ParameterString`, puis déterminez si la limite est imposée par le SDK ou par SageMaker. Le travail est terminé lorsque les arguments de step pris en charge peuvent dépasser 256 caractères tout en conservant une contrainte sûre sur la longueur globale de la commande, avec des tests couvrant des valeurs jusqu’à la limite choisie.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- python
- Domaine
- machine-learning
- Type d'issue
- Fonctionnalité
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- À l'abandon
- Clarté
- Plutôt claire
- Accessibilité débutants
- 35/100