aws / aws/aws-step-functions-data-science-sdk-python
different name for training job inside estimator than step input
- Lingua principale
- Python
- Stelle
- 299
- Fork
- 84
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
- sagemaker contrainer : conda_pytorch_p36
- estimator mode : 'script mode'
While it is a MUST param that I have to give a name for `TrainingJobName` from step functions for data science sdk.
```
pytorch_estimator = PyTorch(entry_point='HRC_0818_final.py',
train_instance_type='ml.m4.xlarge',
role=role,
train_instance_count=1,
framework_version='1.4.0',
base_job_name = 'kanto-base-job',
)
```
```
import stepfunctions
training_step = steps.TrainingStep(
'Model Training',
estimator=pytorch_estimator,
data={
'training': s3_input(s3_data=execution_input['TrainTargetLocation'])
} ,
job_name=execution_input['TrainingJobName'],
wait_for_completion=True
)
model_step = steps.ModelStep(
'Save model',
model=training_step.get_expected_model(),
model_name=execution_input['ModelName'] ,
instance_type='ml.m4.xlarge',
)
execution = workflow.execute(
inputs={
'ModelName': 'kanto-mode-{}'.format(uuid.uuid4().hex),
'TrainTargetLocation' : 's3://hrms-train/traindata/train.jsonl'
}
)
```
it is still the default training job name inside the estimator with current `strtime` following` base_job_name`
` "module_dir": "s3://sagemaker-{aws-region}-{aws-id}/{training-job-name}/source/sourcedir.tar.gz",`
Then you link a wrong dir to a Model consequently.
`SAGEMAKER_SUBMIT_DIRECTORY | s3://sagemaker-{aws-region}-{aws-id}/{base-job-name}-2020-08-20-17-47-50-751/source/sourcedir.tar.gz`
I guess the reason is that I have two difference folder for model.tar.gz and sourcedir.tar.gz then leads to a awkward behavior that you can't create consolidated model.tar.gz when you deploy it to server. I can only copy sourcedir.tar.gz to a mms server as this is a default job name. I am missing model.pth consequently.
So, that just leads to put a lambda function that just copies model.tar.gz (model.pth) from TrainTargetLocation folder to default training job folder (strtime named) to make it work correctly.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia tracciando il modo in cui TrainingStep gestisce job_name insieme a base_job_name dell’estimatore, quindi esamina come vengono costruiti i percorsi di model.tar.gz e sourcedir.tar.gz. Riproduci il comportamento con gli snippet PyTorch, TrainingStep e ModelStep; il lavoro è completato quando il nome del job di training e le posizioni degli artefatti rimangono coerenti senza il workaround di copia segnalato.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- aws, python
- Ambito
- cloud, machine-learning
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Ferma
- Chiarezza
- Da chiarire
- Idoneità per principianti
- 30/100