aws / aws/sagemaker-python-sdk
tensorflow: sm_drivers directory not found
- Langage dominant
- Python
- Étoiles
- 2.3k
- Forks
- 1.3k
- Merge moyen
- 1 j 22 h
- PR mergées (30 j)
- 35
Description
**PySDK Version**
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)
**Describe the bug**
When running on the latest Tensorflow CPU training image: `763104351884.dkr.ecr.eu-central-1.amazonaws.com/tensorflow-training:2.19-cpu-py312`, the job cannot properly run, because the `sm_drivers` directory seems to not be present on the container image (see logs).
**To reproduce**
```python
from pathlib import Path
from sagemaker.train.configs import Compute, InputData, SourceCode
from sagemaker.train.model_trainer import ModelTrainer
EXPERIMENT_NAME = "my-experiment"
ROLE_ARN = "arn:aws:iam::111222333444:role/SageMakerS3AccessRole"
TRAINING_IMAGE = (
"763104351884.dkr.ecr.eu-central-1.amazonaws.com/tensorflow-training:2.19-cpu-py312"
)
INSTANCE_TYPE = "ml.c5.4xlarge"
DATASET_PATH = "s3://my-bucket/data/"
if __name__ == "__main__":
trainer = ModelTrainer(
base_job_name=EXPERIMENT_NAME,
role=ROLE_ARN,
training_image=TRAINING_IMAGE,
source_code=SourceCode(
source_dir=str(Path(__file__).parent),
entry_script="train.py",
requirements="requirements.sagemaker.txt",
ignore_patterns=[
"data",
".venv",
"notebooks",
"environment",
"scripts",
"pipelines",
".ipynb_checkpoints",
".github",
"__pycache__",
"*.ipynb",
],
),
compute=Compute(
instance_type=INSTANCE_TYPE
),
)
input_data = InputData(
channel_name="training",
data_source=DATASET_PATH,
)
trainer.train(input_data_config=[input_data])
```
**Expected behavior**
The job runs.
**Screenshots or logs**
See logs:
```
{"code":{"TrainingInputMode":"File","S3DistributionType":"FullyReplicated","RecordWrapperType":"None"},"sm_drivers":{"TrainingInputMode":"File","S3DistributionType":"FullyReplicated","RecordWrapperType":"None"},"training":{"
TrainingInputMode":"File","S3DistributionType":"FullyReplicated","RecordWrapperType":"None"}}
++ /usr/local/bin/python3 /opt/ml/input/data/sm_drivers/scripts/environment.py
Setting up environment variables
/usr/local/bin/python3: can't open file '/opt/ml/input/data/sm_drivers/scripts/environment.py': [Errno 2] No such file or directory
```
**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**:
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: tensorflow
- **Framework version**: 2.19
- **Python version**: 3.12
- **CPU or GPU**: CPU
- **Custom Docker image (Y/N)**: N
**Additional context**
Add any other context about the problem here.
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Start by reproducing the supplied ModelTrainer example with the TensorFlow 2.19 CPU image and inspect the logged invocation of /opt/ml/input/data/sm_drivers/scripts/environment.py. Trace how the training input and sm_drivers directory are prepared, then verify that the job runs successfully with the affected image.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- aws, python, tensorflow
- Domaine
- machine-learning
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- À l'abandon
- Clarté
- Plutôt claire
- Accessibilité débutants
- 38/100