aws / aws/sagemaker-python-sdk

ModelTrainer generates sm_train.sh with CRLF line endings on Windows causing training job failure

Aperta Adatta ai principianti
#5,904 0 commenti 1 reazione 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
2.3k
Fork
1.3k
Merge medio
1g 22h
PR unite (30g)
35

Descrizione

**PySDK Version**
- [ ] PySDK V3 (3.x)

**Describe the bug**
When using `ModelTrainer` with `SourceCode` on Windows, the SDK internally generates `sm_train.sh` with CRLF (`\r\n`) line endings. This causes the training job to fail immediately when the Linux container tries to execute it.

The root cause is in model_trainer.py in the _prepare_train_script method:
with open(os.path.join(tmp_dir.name, TRAIN_SCRIPT), "w") as f:
f.write(train_script)

**To reproduce**
from sagemaker.train import ModelTrainer
from sagemaker.train.configs import SourceCode, Compute, InputData, OutputDataConfig

source_code = SourceCode(
source_dir="src",
entry_script="train.py",
requirements="requirements.txt"
)

compute = Compute(
instance_type="ml.m5.xlarge",
instance_count=1
)

model_trainer = ModelTrainer(
training_image="",
role="",
source_code=source_code,
compute=compute,
)

train_data = InputData(channel_name="train", data_source="s3://bucket/train/")
val_data = InputData(channel_name="validation", data_source="s3://bucket/val/")

model_trainer.train(input_data_config=[train_data, val_data], wait=True)

**Expected behavior**
sm_train.sh should always be written with LF (\n) line endings regardless of the host OS, since it will always be executed inside a Linux container.

**Error in CloudWatch Logs**
/opt/ml/input/data/sm_drivers/sm_train.sh: line 1: $'\r': command not found
/opt/ml/input/data/sm_drivers/sm_train.sh: line 3: set: -#015: invalid option
set: usage: set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]
/opt/ml/input/data/sm_drivers/sm_train.sh: line 6: syntax error near unexpected token `$'{\r''
/opt/ml/input/data/sm_drivers/sm_train.sh: line 6: `handle_error() {#015'

**Proposed Fix**
## Current code (line in _prepare_train_script):
### with open(os.path.join(tmp_dir.name, TRAIN_SCRIPT), "w") as f:

## Fix — force LF line endings:
### with open(os.path.join(tmp_dir.name, TRAIN_SCRIPT), "w", newline="\n") as f:

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**:3.x
- **Framework name or algorithm **:XgBoost
- **Framework version**:1.7-1
- **Python version**:3.11.5
- **CPU or GPU**:CPU (ml.m5.xlarge)
- **Custom Docker image (Y/N)**:N

**Additional context**
This issue affects all Windows users of the new ModelTrainer API (PySDK V3). The sm_train.sh file is generated entirely by the SDK on the client machine and is never touched by the user, making it impossible to fix without either patching the SDK or switching to the older Estimator API. The fix is a single-character change adding newline="\n" to the open() call.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Aprire model_trainer.py e ispezionare il metodo _prepare_train_script, dove viene scritto sm_train.sh. Verificare che lo script generato utilizzi terminatori di riga LF su Windows, quindi eseguire i controlli pertinenti di ModelTrainer oppure riprodurre il percorso di generazione Windows per confermare che lo script possa essere eseguito nel container Linux.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
aws, python
Ambito
machine-learning
Tipo di issue
Bug
Difficoltà
1/5
Tempo stimato
Meno di un'ora
Stato di attività
Tranquilla
Chiarezza
Specificata chiaramente
Idoneità per principianti
88/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.