aws / aws/sagemaker-python-sdk

ModelTrainer doesn't propagate hyperparameters if SourceCode-command is used

Aperta
#5,226 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
component: training type: bug
Lingua principale
Python
Stelle
2.3k
Fork
1.3k
Merge medio
1g 22h
PR unite (30g)
35

Descrizione

**Describe the bug**
With ModelTrainer, when I'm using the command parameter in the SourceCode with an argument provided as part of the command, for example `python launcher.py -e test.py`, hyperparameters defined in the ModelTrainer are not passed to the training script.

**To reproduce**
A clear, step-by-step set of instructions to reproduce the bug.

```
from sagemaker.modules.configs import (
Compute,
OutputDataConfig,
RemoteDebugConfig,
SourceCode,
StoppingCondition,
)
from sagemaker.modules.train import ModelTrainer

# Define the script to be run
source_code = SourceCode(
source_dir="./scripts",
requirements="requirements.txt",
command="python launcher.py -e train.py",
)

# Define the compute
compute_configs = Compute(
instance_type=instance_type,
instance_count=instance_count,
keep_alive_period_in_seconds=0,
)

job_name = "train-ray-processing-train"

output_path = f"s3://{bucket_name}/{job_name}"

model_trainer = ModelTrainer(
training_image=image_uri,
source_code=source_code,
base_job_name=job_name,
compute=compute_configs,
hyperparameters={
"epochs": 25,
"learning_rate": 0.001,
"batch_size": 100,
},
stopping_condition=StoppingCondition(max_runtime_in_seconds=18000),
output_data_config=OutputDataConfig(
s3_output_path=output_path, compression_type="NONE"
),
role=role,
)
```

in the launcher.py:

```
from argparse import ArgumentParser, Namespace

def __read_params():
try:
parser = ArgumentParser()

parser.add_argument("-e", "--entrypoint", type=str)
parser.add_argument("--epochs", type=int, default=25)
parser.add_argument("--learning_rate", type=float, default=0.001)
parser.add_argument("--batch_size", type=int, default=100)

# Parse only the arguments we care about and ignore the rest
args, unknown = parser.parse_known_args()

return args, unknown
except Exception as e:

raise e

if __name__ == "__main__":
args, _ = __read_params()
```
**Expected behavior**
both the arguments passed as command in the SoureCode, and the hyperparameters provided in the ModelTrainer definition, should be passed to the training script

**Screenshots or logs**
If applicable, add screenshots or logs to help explain your problem.

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: 2.247.1
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: Any
- **Framework version**: Any
- **Python version**: 3.12
- **CPU or GPU**: CPU and GPU
- **Custom Docker image (Y/N)**: N

**Additional context**
Add any other context about the problem here.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia tracciando il modo in cui ModelTrainer combina gli iperparametri con il comando SourceCode, utilizzando la riproduzione in launcher.py e i punti di ingresso ModelTrainer e SourceCode dell'SageMaker SDK. Riproduci il problema con la configurazione fornita, quindi verifica che sia gli argomenti del comando sia gli iperparametri di ModelTrainer arrivino allo script di training.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
aws, python
Ambito
cloud, machine-learning
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.