aws / aws/sagemaker-python-sdk

ModelBuilder.deploy_local() does not authenticate to ECR before pulling DLC images (v3 regression)

クローズ
#5,925 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
2.3k
フォーク
1.3k
平均マージ
1日 22時間
マージ済み PR(30日)
35

説明

## Summary

In v3, `ModelBuilder(mode=Mode.LOCAL_CONTAINER).deploy_local()` calls `docker.client.images.pull(image)` directly with no auth handshake. When the target image lives in an AWS Deep Learning Containers ECR account (`763104351884.dkr.ecr..amazonaws.com/...`) — i.e. *every* SageMaker-provided container — the Docker daemon has no credentials, the pull fails, and the subsequent `inspect_image` returns 404, surfacing as:

```
ValueError: Could not find image '763104351884.dkr.ecr.us-east-1.amazonaws.com/sagemaker-tritonserver:24.09-py3' in repository
```

This is a regression from v2 `LocalSession`/`local.image`, which performed `aws ecr get-login` before pulling.

## Reproduction

```python
from sagemaker.serve import ModelBuilder, Mode, ModelServer

builder = ModelBuilder(
image_uri="763104351884.dkr.ecr.us-east-1.amazonaws.com/sagemaker-tritonserver:24.09-py3",
s3_model_data_url="s3://.../model.tar.gz",
role_arn=role,
model_server=ModelServer.TRITON,
mode=Mode.LOCAL_CONTAINER,
)
builder.build(model_name="x")
builder.deploy_local(endpoint_name="x", wait=True)
```

On a host where Docker has not previously authenticated to the DLC ECR account (e.g. a fresh SageMaker Notebook instance with Docker installed but unused), the pull fails.

## Observed

```
HTTPError: 404 Client Error: Not Found for url:
http+docker://localhost/v1.44/images/763104351884.dkr.ecr.us-east-1.amazonaws.com/sagemaker-tritonserver:24.09-py3/json
...
ImageNotFound: 404 Client Error ... ("No such image: ...")
...
ValueError: Could not find image '...' in repository
```

The `images.pull()` call upstream silently failed (it returns a 404-on-inspect rather than raising for the actual pull-time auth error), so the user-facing error blames "image not found" when the real cause is "Docker daemon was never told how to authenticate to this ECR registry."

## Expected

Either:

1. `deploy_local()` performs an `aws ecr get-login-password | docker login` for any ECR-flavored `image_uri` before calling `images.pull()`, mirroring v2 behavior; **or**
2. `deploy_local()` raises a clear, actionable error explaining the user must pre-authenticate Docker to ECR, with the exact command shown.

## Code reference

`sagemaker/serve/mode/local_container_mode.py:270-278`:

```python
# Pull the image
try:
logger.info("Pulling image %s from repository...", image)
self.client.images.pull(image)
logger.info("Successfully pulled image %s", image)
except docker.errors.NotFound as e:
raise ValueError(f"Could not find image '{image}' in repository") from e
except docker.errors.APIError as e:
raise RuntimeError(f"Failed to pull image '{image}': {e}") from e
```

No `auth_config=` passed to `images.pull()`, no ECR token retrieval, no detection of `*.dkr.ecr.*.amazonaws.com` hostnames.

## Workaround

Pre-authenticate Docker and pre-pull the image before calling `deploy_local()`:

```bash
aws ecr get-login-password --region \
| docker login --username AWS --password-stdin 763104351884.dkr.ecr..amazonaws.com

docker pull 763104351884.dkr.ecr..amazonaws.com/sagemaker-tritonserver:24.09-py3
```

Once the image is in the local Docker cache, the broken `images.pull()` call becomes effectively a no-op and `deploy_local()` proceeds.

## Severity

Medium. Functionally blocks v3 local mode for any AWS-published container image out of the box, which is the most common image source for users. Workaround is mechanical but undocumented in the v3 inference docs.

## Suggestion

Port the `_ecr_login_if_needed` helper from v2's `sagemaker.local.image` (which detects ECR hostnames and runs the login automatically) and call it from `local_container_mode.py:_pull_image()` before `images.pull()`.

## Environment

- **OS:** Linux 6.1.170-210.320.amzn2023.x86_64 (Amazon Linux 2023)
- **Host:** SageMaker Notebook instance (BaseNotebookInstanceEc2InstanceRole)
- **Python:** 3.10 (`/home/ec2-user/anaconda3/envs/python3/bin/python`)
- **Kernel:** `conda_python3`
- **sagemaker:** 3.12.0
- **sagemaker-core:** 2.12.0
- **sagemaker-serve:** 1.12.0
- **sagemaker-train:** 1.12.0
- **sagemaker-mlops:** 1.12.0
- **docker (client):** 25.0.14
- **Region:** us-east-1

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

sagemaker/serve/mode/local_container_mode.py:270-278 から始めて _pull_image() を調査し、その後 v2's sagemaker.local.image の ECR 認証動作と比較します。ローカルデプロイが SageMaker イメージを pull する前に ECR で認証するか、イメージが見つからないと誤って報告するのではなく、対処可能な事前認証エラーを報告すれば完了です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
aws, docker, python
領域
cloud, devops, machine-learning
issue の種類
バグ
難易度
3/5
見積もり時間
1〜2日
活発さ
静か
明瞭さ
明確に書かれている
初心者へのやさしさ
76/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。