aws / aws/sagemaker-python-sdk

ModelBuilder.deploy_local() does not authenticate to ECR before pulling DLC images (v3 regression)

Chiusa
#5,925 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
2.3k
Fork
1.3k
Merge medio
1g 22h
PR unite (30g)
35

Descrizione

## Summary

In v3, `ModelBuilder(mode=Mode.LOCAL_CONTAINER).deploy_local()` calls `docker.client.images.pull(image)` directly with no auth handshake. When the target image lives in an AWS Deep Learning Containers ECR account (`763104351884.dkr.ecr..amazonaws.com/...`) — i.e. *every* SageMaker-provided container — the Docker daemon has no credentials, the pull fails, and the subsequent `inspect_image` returns 404, surfacing as:

```
ValueError: Could not find image '763104351884.dkr.ecr.us-east-1.amazonaws.com/sagemaker-tritonserver:24.09-py3' in repository
```

This is a regression from v2 `LocalSession`/`local.image`, which performed `aws ecr get-login` before pulling.

## Reproduction

```python
from sagemaker.serve import ModelBuilder, Mode, ModelServer

builder = ModelBuilder(
image_uri="763104351884.dkr.ecr.us-east-1.amazonaws.com/sagemaker-tritonserver:24.09-py3",
s3_model_data_url="s3://.../model.tar.gz",
role_arn=role,
model_server=ModelServer.TRITON,
mode=Mode.LOCAL_CONTAINER,
)
builder.build(model_name="x")
builder.deploy_local(endpoint_name="x", wait=True)
```

On a host where Docker has not previously authenticated to the DLC ECR account (e.g. a fresh SageMaker Notebook instance with Docker installed but unused), the pull fails.

## Observed

```
HTTPError: 404 Client Error: Not Found for url:
http+docker://localhost/v1.44/images/763104351884.dkr.ecr.us-east-1.amazonaws.com/sagemaker-tritonserver:24.09-py3/json
...
ImageNotFound: 404 Client Error ... ("No such image: ...")
...
ValueError: Could not find image '...' in repository
```

The `images.pull()` call upstream silently failed (it returns a 404-on-inspect rather than raising for the actual pull-time auth error), so the user-facing error blames "image not found" when the real cause is "Docker daemon was never told how to authenticate to this ECR registry."

## Expected

Either:

1. `deploy_local()` performs an `aws ecr get-login-password | docker login` for any ECR-flavored `image_uri` before calling `images.pull()`, mirroring v2 behavior; **or**
2. `deploy_local()` raises a clear, actionable error explaining the user must pre-authenticate Docker to ECR, with the exact command shown.

## Code reference

`sagemaker/serve/mode/local_container_mode.py:270-278`:

```python
# Pull the image
try:
logger.info("Pulling image %s from repository...", image)
self.client.images.pull(image)
logger.info("Successfully pulled image %s", image)
except docker.errors.NotFound as e:
raise ValueError(f"Could not find image '{image}' in repository") from e
except docker.errors.APIError as e:
raise RuntimeError(f"Failed to pull image '{image}': {e}") from e
```

No `auth_config=` passed to `images.pull()`, no ECR token retrieval, no detection of `*.dkr.ecr.*.amazonaws.com` hostnames.

## Workaround

Pre-authenticate Docker and pre-pull the image before calling `deploy_local()`:

```bash
aws ecr get-login-password --region \
| docker login --username AWS --password-stdin 763104351884.dkr.ecr..amazonaws.com

docker pull 763104351884.dkr.ecr..amazonaws.com/sagemaker-tritonserver:24.09-py3
```

Once the image is in the local Docker cache, the broken `images.pull()` call becomes effectively a no-op and `deploy_local()` proceeds.

## Severity

Medium. Functionally blocks v3 local mode for any AWS-published container image out of the box, which is the most common image source for users. Workaround is mechanical but undocumented in the v3 inference docs.

## Suggestion

Port the `_ecr_login_if_needed` helper from v2's `sagemaker.local.image` (which detects ECR hostnames and runs the login automatically) and call it from `local_container_mode.py:_pull_image()` before `images.pull()`.

## Environment

- **OS:** Linux 6.1.170-210.320.amzn2023.x86_64 (Amazon Linux 2023)
- **Host:** SageMaker Notebook instance (BaseNotebookInstanceEc2InstanceRole)
- **Python:** 3.10 (`/home/ec2-user/anaconda3/envs/python3/bin/python`)
- **Kernel:** `conda_python3`
- **sagemaker:** 3.12.0
- **sagemaker-core:** 2.12.0
- **sagemaker-serve:** 1.12.0
- **sagemaker-train:** 1.12.0
- **sagemaker-mlops:** 1.12.0
- **docker (client):** 25.0.14
- **Region:** us-east-1

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia da sagemaker/serve/mode/local_container_mode.py:270-278 e analizza _pull_image(), quindi confronta il comportamento di autenticazione ECR in v2's sagemaker.local.image. Il lavoro è completato quando il deployment locale esegue l'autenticazione a ECR prima di effettuare il pull delle immagini SageMaker, oppure segnala un errore di pre-autenticazione su cui sia possibile intervenire, invece di indicare erroneamente che manca un'immagine.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
aws, docker, python
Ambito
cloud, devops, machine-learning
Tipo di issue
Bug
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Tranquilla
Chiarezza
Specificata chiaramente
Idoneità per principianti
76/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.