aws / aws/sagemaker-python-sdk
tensorflow: sm_drivers directory not found
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 1.3k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 32
Description
**PySDK Version**
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)
**Describe the bug**
When running on the latest Tensorflow CPU training image: `763104351884.dkr.ecr.eu-central-1.amazonaws.com/tensorflow-training:2.19-cpu-py312`, the job cannot properly run, because the `sm_drivers` directory seems to not be present on the container image (see logs).
**To reproduce**
```python
from pathlib import Path
from sagemaker.train.configs import Compute, InputData, SourceCode
from sagemaker.train.model_trainer import ModelTrainer
EXPERIMENT_NAME = "my-experiment"
ROLE_ARN = "arn:aws:iam::111222333444:role/SageMakerS3AccessRole"
TRAINING_IMAGE = (
"763104351884.dkr.ecr.eu-central-1.amazonaws.com/tensorflow-training:2.19-cpu-py312"
)
INSTANCE_TYPE = "ml.c5.4xlarge"
DATASET_PATH = "s3://my-bucket/data/"
if __name__ == "__main__":
trainer = ModelTrainer(
base_job_name=EXPERIMENT_NAME,
role=ROLE_ARN,
training_image=TRAINING_IMAGE,
source_code=SourceCode(
source_dir=str(Path(__file__).parent),
entry_script="train.py",
requirements="requirements.sagemaker.txt",
ignore_patterns=[
"data",
".venv",
"notebooks",
"environment",
"scripts",
"pipelines",
".ipynb_checkpoints",
".github",
"__pycache__",
"*.ipynb",
],
),
compute=Compute(
instance_type=INSTANCE_TYPE
),
)
input_data = InputData(
channel_name="training",
data_source=DATASET_PATH,
)
trainer.train(input_data_config=[input_data])
```
**Expected behavior**
The job runs.
**Screenshots or logs**
See logs:
```
{"code":{"TrainingInputMode":"File","S3DistributionType":"FullyReplicated","RecordWrapperType":"None"},"sm_drivers":{"TrainingInputMode":"File","S3DistributionType":"FullyReplicated","RecordWrapperType":"None"},"training":{"
TrainingInputMode":"File","S3DistributionType":"FullyReplicated","RecordWrapperType":"None"}}
++ /usr/local/bin/python3 /opt/ml/input/data/sm_drivers/scripts/environment.py
Setting up environment variables
/usr/local/bin/python3: can't open file '/opt/ml/input/data/sm_drivers/scripts/environment.py': [Errno 2] No such file or directory
```
**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**:
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: tensorflow
- **Framework version**: 2.19
- **Python version**: 3.12
- **CPU or GPU**: CPU
- **Custom Docker image (Y/N)**: N
**Additional context**
Add any other context about the problem here.
Contributor guide
Assessment
This issue has not been assessed yet.