aws / aws/sagemaker-python-sdk

tensorflow: sm_drivers directory not found

Offen
#5,493 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
2.3k
Forks
1.3k
Ø Merge
1 T. 22 Std.
Gemergte PRs (30 T.)
35

Beschreibung

**PySDK Version**
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)

**Describe the bug**
When running on the latest Tensorflow CPU training image: `763104351884.dkr.ecr.eu-central-1.amazonaws.com/tensorflow-training:2.19-cpu-py312`, the job cannot properly run, because the `sm_drivers` directory seems to not be present on the container image (see logs).

**To reproduce**

```python
from pathlib import Path

from sagemaker.train.configs import Compute, InputData, SourceCode
from sagemaker.train.model_trainer import ModelTrainer

EXPERIMENT_NAME = "my-experiment"
ROLE_ARN = "arn:aws:iam::111222333444:role/SageMakerS3AccessRole"
TRAINING_IMAGE = (
"763104351884.dkr.ecr.eu-central-1.amazonaws.com/tensorflow-training:2.19-cpu-py312"
)
INSTANCE_TYPE = "ml.c5.4xlarge"
DATASET_PATH = "s3://my-bucket/data/"

if __name__ == "__main__":
trainer = ModelTrainer(
base_job_name=EXPERIMENT_NAME,
role=ROLE_ARN,
training_image=TRAINING_IMAGE,
source_code=SourceCode(
source_dir=str(Path(__file__).parent),
entry_script="train.py",
requirements="requirements.sagemaker.txt",
ignore_patterns=[
"data",
".venv",
"notebooks",
"environment",
"scripts",
"pipelines",
".ipynb_checkpoints",
".github",
"__pycache__",
"*.ipynb",
],
),
compute=Compute(
instance_type=INSTANCE_TYPE
),
)

input_data = InputData(
channel_name="training",
data_source=DATASET_PATH,
)

trainer.train(input_data_config=[input_data])
```

**Expected behavior**
The job runs.

**Screenshots or logs**

See logs:
```

{"code":{"TrainingInputMode":"File","S3DistributionType":"FullyReplicated","RecordWrapperType":"None"},"sm_drivers":{"TrainingInputMode":"File","S3DistributionType":"FullyReplicated","RecordWrapperType":"None"},"training":{"
TrainingInputMode":"File","S3DistributionType":"FullyReplicated","RecordWrapperType":"None"}}
++ /usr/local/bin/python3 /opt/ml/input/data/sm_drivers/scripts/environment.py
Setting up environment variables
/usr/local/bin/python3: can't open file '/opt/ml/input/data/sm_drivers/scripts/environment.py': [Errno 2] No such file or directory
```

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**:
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: tensorflow
- **Framework version**: 2.19
- **Python version**: 3.12
- **CPU or GPU**: CPU
- **Custom Docker image (Y/N)**: N

**Additional context**
Add any other context about the problem here.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start by reproducing the supplied ModelTrainer example with the TensorFlow 2.19 CPU image and inspect the logged invocation of /opt/ml/input/data/sm_drivers/scripts/environment.py. Trace how the training input and sm_drivers directory are prepared, then verify that the job runs successfully with the affected image.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
aws, python, tensorflow
Bereich
machine-learning
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
38/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.