aws / aws/sagemaker-python-sdk

Support for docker compose > v2

Closed Beginner friendly
#5,739 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

**PySDK Version**
- [ ] PySDK V2 (2.x)
- [X] PySDK V3 (3.x)

**Describe the bug**
Unable to run sagemaker training in "local" mode with docker version v5.

**Component**: ``sagemaker.train.local.local_container._LocalContainer._get_compose_cmd_prefix``

[_get_compose_cmd_prefix()](https://github.com/aws/sagemaker-python-sdk/blob/91ca011daa8853ee031e784c35a5bab23a79ad08/sagemaker-core/src/sagemaker/core/local/image.py#L153-L168) only recognizes Docker Compose **v2** by
checking ``"v2" in output``. Docker Compose **v5.x** (and v3, v4, or any
future major version) is silently rejected, causing::

ImportError: Docker Compose is not installed.
Local Mode features will not work without docker compose.

even though ``docker compose`` is fully installed and functional.

The substring check ``"v2" in output`` is too narrow. Every Compose
version with a major number other than 2 is treated as "not installed".

**To reproduce**
A clear, step-by-step set of instructions to reproduce the bug.

Required for Test:
docker pull docker:latest
docker tag image to docker:latest

# Make sure the latest version is > 2
docker run --rm docker:latest docker compose version
# Docker Compose version v5.1.1

The provided code need to be **complete** and **runnable**, if additional data is needed, please include them in the issue.

```
import re
import subprocess

import mock

def test_local_train_triggers_compose_v5_bug(tmp_path):
"""Invoke sagemaker local-mode training to prove Docker Compose v5 triggers the bug.

This test exercises the real call chain:
ModelTrainer.train()
→ _LocalContainer.train()
→ _LocalContainer._generate_compose_command()
→ _LocalContainer._get_compose_cmd_prefix() ← bug lives here

The bug: ``_get_compose_cmd_prefix`` only accepts output containing "v2".
Any Compose v3/v4/v5 output causes an ``ImportError`` even though Docker
Compose is fully installed and functional.

Assumes ``docker:latest`` is already pulled on the local workstation.
"""
import os
import boto3
from moto import mock_aws
from sagemaker.core.local import LocalSession
from sagemaker.core.training.configs import Compute, InputData, OutputDataConfig, SourceCode
from sagemaker.train.model_trainer import ModelTrainer, Mode

#Assumes a local docker:latest image is already pulled, which contains a real compose version string in its output. If this test starts failing due to a compose version update, update this image to one that contains a compose version string that triggers the bug (v3+).
PUBLIC_DOCKER_IMAGE = "docker:latest"

# Step 1: get the real compose version string from the local docker:latest image
result = subprocess.run( # nosec B603 B607
["docker", "run", "--rm", PUBLIC_DOCKER_IMAGE, "docker", "compose", "version"],
capture_output=True,
text=True,
timeout=30, # image assumed already pulled — no pull delay
)
assert result.returncode == 0, (
f"Could not get compose version from {PUBLIC_DOCKER_IMAGE}:\n{result.stderr}"
)
real_version_output = result.stdout # e.g. "Docker Compose version v5.1.1\n"

match = re.search(r"v(\d+)", real_version_output.strip())
assert match is not None, f"Could not parse version from: {real_version_output!r}"
major_version = int(match.group(1))

if major_version == 2:
print(
f"docker:latest ships Compose v{major_version} — bug only triggers on v3+. "
"Update PUBLIC_DOCKER_IMAGE to an image that ships Compose v3+."
)

# Step 2: build a minimal LocalSession backed by moto
with mock_aws():
boto_session = boto3.Session(region_name="us-east-1")
s3 = boto_session.client("s3")
s3.create_bucket(Bucket="sagemaker-bug-repro")

sm_session = LocalSession(
boto_session=boto_session,
default_bucket="sagemaker-bug-repro",
sagemaker_config={
"SchemaVersion": "1.0",
"SageMaker": {"PythonSDK": {"Modules": {"TelemetryOptOut": True}}},
},
)
sm_session.config = {
"local": {"local_code": True, "container_root": str(tmp_path)}
}
sm_session._default_bucket = "sagemaker-bug-repro"

# Step 3: build a ModelTrainer pointing at the public docker:latest image.
# We mock subprocess.check_output so the SDK receives the *real* version
# string from the public image instead of whatever is installed on this host.
trainer = ModelTrainer(
training_image=PUBLIC_DOCKER_IMAGE,
role="arn:aws:iam::123456789012:role/fake-role",
compute=Compute(instance_type="local_cpu", instance_count=1),
sagemaker_session=sm_session,
output_data_config=OutputDataConfig(s3_output_path="s3://sagemaker-bug-repro/output"),
base_job_name="compose-v5-bug",
training_mode=Mode.LOCAL_CONTAINER,
local_container_root=str(tmp_path),
source_code=SourceCode(command="true"), # no-op command
)

# Step 4: patch subprocess so _get_compose_cmd_prefix sees the real
# version string from docker:latest (v5.x), then assert the bug fires.
with (
mock.patch("subprocess.check_output", return_value=real_version_output),
mock.patch("shutil.which", return_value=None),
):
trainer.train(wait=False, logs=False)

```

**Expected behavior**
sagemaker train in local mode should not error

**Screenshots or logs --- STACK Trace**
```
.tox/py313/lib/python3.13/site-packages/sagemaker/core/telemetry/telemetry_logging.py:187: in wrapper
raise caught_ex
.tox/py313/lib/python3.13/site-packages/sagemaker/core/telemetry/telemetry_logging.py:153: in wrapper
response = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
.tox/py313/lib/python3.13/site-packages/sagemaker/core/workflow/pipeline_context.py:346: in wrapper
return run_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
.tox/py313/lib/python3.13/site-packages/pydantic/_internal/_validate_call.py:39: in wrapper_function
return wrapper(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
.tox/py313/lib/python3.13/site-packages/pydantic/_internal/_validate_call.py:136: in __call__
res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.tox/py313/lib/python3.13/site-packages/sagemaker/train/model_trainer.py:813: in train
local_container.train(wait)
.tox/py313/lib/python3.13/site-packages/sagemaker/train/local/local_container.py:237: in train
compose_command = self._generate_compose_command(wait)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.tox/py313/lib/python3.13/site-packages/sagemaker/train/local/local_container.py:479: in _generate_compose_command
_compose_cmd_prefix = self._get_compose_cmd_prefix()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = _LocalContainer(training_job_name='a-test-20260408132342', instance_type='local_cpu', instance_count=1, image='glm-sta...er_arguments=['-c', 'chmod +x /opt/ml/input/data/sm_drivers/sm_train.sh && /opt/ml/input/data/sm_drivers/sm_train.sh'])
def _get_compose_cmd_prefix(self) -> List[str]:
"""Gets the Docker Compose command.

The method initially looks for 'docker compose' v2
executable, if not found looks for 'docker-compose' executable.

Returns:
List[str]: Docker Compose executable split into list.

Raises:
ImportError: If Docker Compose executable was not found.
"""
compose_cmd_prefix = []

output = None
try:
output = subprocess.check_output(
["docker", "compose", "version"],
stderr=subprocess.DEVNULL,
encoding="UTF-8",
)
except subprocess.CalledProcessError:
logger.info(
"'Docker Compose' is not installed. "
"Proceeding to check for 'docker-compose' CLI."
)

if output and "v2" in output.strip():
logger.info("'Docker Compose' found using Docker CLI.")
compose_cmd_prefix.extend(["docker", "compose"])
return compose_cmd_prefix

if shutil.which("docker-compose") is not None:
logger.info("'Docker Compose' found using Docker Compose CLI.")
compose_cmd_prefix.extend(["docker-compose"])
return compose_cmd_prefix

> raise ImportError(
"Docker Compose is not installed. "
"Local Mode features will not work without docker compose. "
"For more information on how to install 'docker compose', please, see "
"https://docs.docker.com/compose/install/"
)
E ImportError: Docker Compose is not installed. Local Mode features will not work without docker compose. For more information on how to install 'docker compose', please, see https://docs.docker.com/compose/install/
.tox/py313/lib/python3.13/site-packages/sagemaker/train/local/local_container.py:638: ImportError
```

```
# ═══════════════════════════════════════════════════════════════════════════
# PROPOSED FIX – standalone validation of the corrected logic
# ═══════════════════════════════════════════════════════════════════════════

def _fixed_compose_check(output: str | None) -> bool:
"""Proposed replacement for the ``"v2" in output`` check.

Accepts any Docker Compose plugin version >= 2.0.0.

Examples
--------
>>> _fixed_compose_check("Docker Compose version v5.1.1")
True
>>> _fixed_compose_check("Docker Compose version v2.27.0")
True
>>> _fixed_compose_check("Docker Compose version v1.29.2")
False
>>> _fixed_compose_check("")
False
>>> _fixed_compose_check(None)
False
"""
if not output:
return False
match = re.search(r"v(\d+)", output.strip())
return match is not None and int(match.group(1)) >= 2
```

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: sagemaker-train (>=1.6.0,<2.0.0) / sagemaker-core (>=2.7.1,<3.0.0)
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: NA
- **Framework version**: sagemaker-train (>=1.6.0,<2.0.0) / sagemaker-core (>=2.7.1,<3.0.0)
- **Python version**: 3.13
- **CPU or GPU**: CPU
- **Custom Docker image (Y/N)**: N

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Research direction

Start at sagemaker/train/local/local_container.py and inspect _get_compose_cmd_prefix(), which currently checks the Docker Compose version output. Add focused coverage for supported Compose major versions and the fallback CLI, then run the local training tests; done means Docker Compose v2 and later work without an ImportError while unsupported or unavailable commands still follow the fallback path.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
devops, machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.