aws / aws/sagemaker-python-sdk
Sagemaker sdk does not support un-versioned docker API's (Sagemaker Studio)
- 主要言語
- Python
- スター
- 2.3k
- フォーク
- 1.3k
- 平均マージ
- 1日 22時間
- マージ済み PR(30日)
- 35
説明
**Describe the bug**
I'm encountering an issue while attempting to build a local model deployment using the ModelBuilder class in SageMaker Studio. When I attempt to use ModelBuilder, I get the following error:
`DockerException: Error while fetching server API version: Expecting value: line 1 column 1 (char 0)
`The problem seems to stem from the fact that the Docker version in SageMaker Studio is not recognized, which causes this error when trying to interact with Docker as part of the local deployment process. This impacts the usage of the ModelBuilder class, as it is unable to retrieve necessary Docker API information.
Better explained issue with mlflow:
https://github.com/orgs/aws/projects/160/views/1?filterQuery=docker&pane=issue&itemId=95146971&issue=aws%7Camazon-sagemaker-feedback%7C161.
By changing directly in sdk docker.from_env() to docker.from_env(version='1.44') solves this issue, but then there is another issue with local mdoe:
```
Forbidden ("Forbidden. Reason:
[ContainerCreate] SageMaker Studio Docker Usage only allows 'sagemaker' as network input")
```
**To reproduce**
In sagemaker studio:
```
model_name = custom
model_version = 1
MLFLOW_TRACKING_ARN = "your tracking uri"
execution_role = get_execution_role() if get_execution_role() is not None else "your-role-arn"
model_builder = ModelBuilder(
mode=Mode.LOCAL_CONTAINER,
schema_builder=model_schema,
role_arn=execution_role,
model_metadata={
"MLFLOW_MODEL_PATH": f"models:/{model_name}/{model_version}",
"MLFLOW_TRACKING_ARN" : '/home/sagemaker-user/projects/clvm/mlruns'
})
```
**Expected behavior**
Are able to deploy a model locally/on cloud.
**Screenshots or logs**
```
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/docker/api/client.py:223 │
│ in _retrieve_server_version │
│ │
│ 220 │ │
│ 221 │ def _retrieve_server_version(self): │
│ 222 │ │ try: │
│ ❱ 223 │ │ │ return self.version(api_version=False)["ApiVersion"] │
│ 224 │ │ except KeyError as ke: │
│ 225 │ │ │ raise DockerException( │
│ 226 │ │ │ │ 'Invalid response from docker daemon: key "ApiVersion"' │
│ │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/docker/api/daemon.py:181 │
│ in version │
│ │
│ 178 │ │ │ │ If the server returns an error. │
│ 179 │ │ """ │
│ 180 │ │ url = self._url("/version", versioned_api=api_version) │
│ ❱ 181 │ │ return self._result(self._get(url), json=True) │
│ 182 │
│ │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/docker/api/client.py:284 │
│ in _result │
│ │
│ 281 │ │ self._raise_for_status(response) │
│ 282 │ │ │
│ 283 │ │ if json: │
│ ❱ 284 │ │ │ return response.json() │
│ 285 │ │ if binary: │
│ 286 │ │ │ return response.content │
│ 287 │ │ return response.text │
│ │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/requests/models.py:978 in │
│ json │
│ │
│ 975 │ │ except JSONDecodeError as e: │
│ 976 │ │ │ # Catch JSON-related errors and raise as requests.JSONDecodeError │
│ 977 │ │ │ # This aliases json.JSONDecodeError and simplejson.JSONDecodeError │
│ ❱ 978 │ │ │ raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) │
│ 979 │ │
│ 980 │ @property │
│ 981 │ def links(self): │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in :1 │
│ │
│ ❱ 1 predictor = model.deploy( │
│ 2 ) │
│ 3 │
│ │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/sagemaker/serve/utils/tele │
│ metry_logger.py:116 in wrapper │
│ │
│ 113 │ │ │ │ failure_reason = str(e) │
│ 114 │ │ │ │ failure_type = e.__class__.__name__ │
│ 115 │ │ │ except Exception as e: # pylint: disable=W0703 │
│ ❱ 116 │ │ │ │ raise e │
│ 117 │ │ │ │
│ 118 │ │ │ stop_timer = perf_counter() │
│ 119 │ │ │ elapsed = stop_timer - start_timer │
│ │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/sagemaker/serve/utils/tele │
│ metry_logger.py:104 in wrapper │
│ │
│ 101 │ │ │ │
│ 102 │ │ │ start_timer = perf_counter() │
│ 103 │ │ │ try: │
│ ❱ 104 │ │ │ │ response = func(self, *args, **kwargs) │
│ 105 │ │ │ except ( │
│ 106 │ │ │ │ ModelBuilderException, │
│ 107 │ │ │ │ exceptions.CapacityError, │
│ │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/sagemaker/serve/builder/mo │
│ del_builder.py:600 in _model_builder_deploy_wrapper │
│ │
│ 597 │ │ │ │ deserializer=deserializer, │
│ 598 │ │ │ ) │
│ 599 │ │ │ │
│ ❱ 600 │ │ │ self.modes[str(Mode.LOCAL_CONTAINER)].create_server( │
│ 601 │ │ │ │ self.image_uri, container_timeout_in_second, self.secret_key, predictor │
│ 602 │ │ │ ) │
│ 603 │ │ │ return predictor │
│ │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/sagemaker/serve/mode/local │
│ _container_mode.py:100 in create_server │
│ │
│ 97 │ ): │
│ 98 │ │ """Placeholder docstring""" │
│ 99 │ │ │
│ ❱ 100 │ │ self._pull_image(image=image) │
│ 101 │ │ │
│ 102 │ │ self.destroy_server() │
│ 103 │
│ │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/sagemaker/serve/mode/local │
│ _container_mode.py:232 in _pull_image │
│ │
│ 229 │ │ except subprocess.CalledProcessError as e: │
│ 230 │ │ │ logger.warning("Unable to login to ecr: %s", e) │
│ 231 │ │ │
│ ❱ 232 │ │ self.client = docker.from_env() │
│ 233 │ │ try: │
│ 234 │ │ │ logger.info("Pulling image %s from repository...", image) │
│ 235 │ │ │ self.client.images.pull(image) │
│ │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/docker/client.py:94 in │
│ from_env │
│ │
│ 91 │ │ max_pool_size = kwargs.pop('max_pool_size', DEFAULT_MAX_POOL_SIZE) │
│ 92 │ │ version = kwargs.pop('version', None) │
│ 93 │ │ use_ssh_client = kwargs.pop('use_ssh_client', False) │
│ ❱ 94 │ │ return cls( │
│ 95 │ │ │ timeout=timeout, │
│ 96 │ │ │ max_pool_size=max_pool_size, │
│ 97 │ │ │ version=version, │
│ │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/docker/client.py:45 in │
│ __init__ │
│ │
│ 42 │ │ │ to save in the pool. │
│ 43 │ """ │
│ 44 │ def __init__(self, *args, **kwargs): │
│ ❱ 45 │ │ self.api = APIClient(*args, **kwargs) │
│ 46 │ │
│ 47 │ @classmethod │
│ 48 │ def from_env(cls, **kwargs): │
│ │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/docker/api/client.py:207 │
│ in __init__ │
│ │
│ 204 │ │ │ │ │ │ │ │ version, │
│ 205 │ │ │ │ │ │ │ │ str │
│ 206 │ │ │ │ │ │ │ │ ) and version.lower() == 'auto'): │
│ ❱ 207 │ │ │ self._version = self._retrieve_server_version() │
│ 208 │ │ else: │
│ 209 │ │ │ self._version = version │
│ 210 │ │ if not isinstance(self._version, str): │
│ │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/docker/api/client.py:230 │
│ in _retrieve_server_version │
│ │
│ 227 │ │ │ │ ' is missing.' │
│ 228 │ │ │ ) from ke │
│ 229 │ │ except Exception as e: │
│ ❱ 230 │ │ │ raise DockerException( │
│ 231 │ │ │ │ f'Error while fetching server API version: {e}' │
│ 232 │ │ │ ) from e │
│ 233 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
DockerException: Error while fetching server API version: Expecting value: line 1 column 1 (char 0)
```**System information**
A description of your system. Please provide:
- sagemaker==2.242.2
- mlflow==2.21.0
- python 3.11.11
- cpu
- No custom docker image
- SageMaker Distribution 3.0.0 image in sagemaker studio
コントリビューションガイド
調査の方向性
sagemaker/serve/mode/local_container_mode.py から始め、特に traceback が docker.from_env() に到達する create_server と _pull_image を確認します。SageMaker Studio で ModelBuilder のローカルコンテナフローを再現し、Studio Docker API のレスポンスとその許可されたネットワーク設定でデプロイが機能することを確認します。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- aws, docker, python
- 領域
- devops, machine-learning
- issue の種類
- バグ
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 停滞
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 35/100