aws / aws/sagemaker-python-sdk

Increase Length Limit for Arguments to Steps

Aperta
#5,224 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
component: processing type: feature request
Lingua principale
Python
Stelle
2.3k
Fork
1.3k
Merge medio
1g 22h
PR unite (30g)
35

Descrizione

**Describe the feature you'd like**
Consider raising the character limit on arguments passed to scripts a little. Right now, each parameter passed to a step through `job_arguments` must be no more than 256 characters. This is an oppressive limit. Consider:

```
arguments = [
"--s3_data_path",
s3_path,
"--model_description",
# This, whether it's Parameter or str, cannot be > 256 chars
model_description,
]

preprocessing_step = ProcessingStep(
name=...,
processor=...,
code=...,
job_arguments=arguments,
)
```

As a result, when building pipelines, you end up having to upload even something simple like a description string to a file on S3. What's worse, is that if `model_description` in the example above is a `Parameter`, a `Parameter` can legally be up to 1024 characters. So not even every Parameter can be passed to a Step! Whether the above code will break or not, if `model_description` is a `Parameter`, depends on the Parameter's runtime value. Surely this is a very sad state of affairs.

Even just fairly modest increases to the size limits would greatly improve quality-of-life. At the very least, the size limit should match the size limit of ParameterString, at 1024 characters, so that any `Parameter` can be passed as a step argument. Ideally, for both ParameterString and the argument length, the limit would be increased further to a number more like 10k. That would allow any small content that's reasonable to have as a Parameter (e.g. small text blob) to be a Parameter while things like data files, etc. would still have to be uploaded to S3.

I am fairly confident that this could be done without hitting command length limitations on any modern shell. If the issue is that there may be up to 100 parameters, impose an _overall_ length limit on the shell command so that each individual parameter does not need to be so constrained.

**How would this feature be used? Please describe.**
Users would more easily pass more data directly, without having to put things in S3 files all the time, greatly increasing the usability of Sagemaker pipelines.

**Describe alternatives you've considered**
The workaround is that any content even potentially exceeding 256 characters must go in an S3 file.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia individuando la validazione di `job_arguments` e la gestione della lunghezza di `ParameterString`, quindi determina se il limite è imposto dall'SDK o da SageMaker. Il lavoro è completato quando gli argomenti di step supportati possono superare i 256 caratteri mantenendo un vincolo sicuro sulla lunghezza complessiva del comando, con test che coprano valori fino al limite scelto.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
machine-learning
Tipo di issue
Funzionalità
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.