aws / aws/amazon-sagemaker-feedback

Long wait on "INFO: Invoking remote function inside conda environment: sagemaker-runtime-env" when executing training job

Open
#83 0 comments 0 reactions 0 assignees View on GitHub
bug training
Dominant language
No language data
Stars
10
Forks
3
PR merge metrics
No merged PRs in 30d

Description

### Product Version

- [ ] Amazon SageMaker Studio Classic
- [X] Amazon SageMaker Studio
- [ ] Issue is not related to SageMaker Studio

### Issue Description

The problem occurs when I create a training job using Sagemaker Python SDK.
```
with RemoteExecutor(instance_type="ml.g4dn.2xlarge", dependencies='./timeseries_env.yml', max_parallel_jobs=1, keep_alive_period_in_seconds=30) as executor:
future = executor.submit(training_job, arg1, arg2)
```
After dependencies are installed from a yaml file (content provided below) the job freezes for more than an hour on the following line: INFO: Invoking remote function inside conda environment: sagemaker-runtime-env.

```
name: my_env
channels:
- conda-forge
- defaults
dependencies:
- python=3.10
- pandas
- scikit-learn
- matplotlib
- pip:
- sagemaker
- s3fs
- fsspec
- neuralforecast
- hyperopt
```

### Expected Behavior

The job doesn't freeze for so long on the mentioned line

### Observed Behavior

_No response_

### Product Category

Jobs

### Feedback Category

Customer Support, Reliability and Stability, Startup Time and Latency

### Other Details

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.