When will an AutoGluon 1.6.x training DLC be published?

# When will an AutoGluon 1.6.x training DLC be published?

## Summary

The newest AutoGluon Deep Learning Container available for SageMaker training is **1.5.0**, while the current AutoGluon release on PyPI is **1.6.1**. This version gap makes `autogluon.cloud` unusable for any project that runs 1.6.x locally, because a cloud-trained predictor can neither be produced with 1.6.x-only settings nor loaded afterwards by a 1.6.x installation.

I would like to know whether a 1.6.x training container is planned, and if so, roughly when it is expected to be published.

## Environment

| | |
|---|---|
| Local AutoGluon | 1.6.1 |
| `autogluon.cloud` | 0.5.0 |
| boto3 / botocore | 1.43.66 |
| Python | 3.12 |
| Region | `eu-central-1` |
| Instance type | `ml.g4dn.2xlarge` |

## What `latest` resolves to

```python
>>> from autogluon.cloud.utils.dlc_utils import retrieve_available_framework_versions, retrieve_latest_framework_version
>>> retrieve_available_framework_versions("training")
['0.3.1', '0.3.2', '0.4.0', '0.4.2', '0.4.3', '0.5.2', '0.6.1', '0.6.2',
'0.7.0', '0.8.2', '1.0.0', '1.1.0', '1.1.1', '1.2.0', '1.3.0', '1.4.0', '1.5.0']
>>> retrieve_latest_framework_version("training")
('1.5.0', ['py312'])
```

The resulting image for a GPU training job:

```
763104351884.dkr.ecr.eu-central-1.amazonaws.com/autogluon-training:1.5.0-gpu-py312-cu126-ubuntu22.04
```

There is no 1.6.x tag to pin instead, so `framework_version="latest"` is already the best available choice.

## Concrete problems this causes

### 1. Presets introduced in 1.6.x are rejected by the 1.5.0 container

A `TimeSeriesCloudPredictor.fit(...)` call using the `experimental` preset fails inside the container:

```
AlgorithmError: ExecuteUserScriptError:
ExitCode 1
ErrorMessage "raise ValueError(f"Preset '{preset_og}' was not found. Valid presets: {sorted(set(valid_presets))}")
ValueError: Preset 'experimental' was not found. Valid presets: ['best', 'best_quality',
'bolt_base', 'bolt_mini', 'bolt_small', 'bolt_tiny', 'bq', 'chronos2', 'chronos2_ensemble',
'chronos2_small', 'fast_training', 'high', 'high_quality', 'hq', 'medium', 'medium_quality', 'mq']"
Command "/usr/local/bin/python train.py", exit code: 1
```

Comparing the TimeSeries preset registries confirms the gap is exactly the two 1.6.x additions:

```
local 1.6.1 only : ['experimental', 'experimental_quality']
container 1.5.0 only : []
common : the remaining 17 presets
```

The failure only surfaces after the instance has been provisioned and the training image downloaded, so the GPU time is already billed before the job exits.

### 2. A predictor trained in the container cannot be loaded locally

`TabularPredictor.load()` and `TimeSeriesPredictor.load()` both default to `require_version_match=True`, and the underlying check compares versions for exact string equality:

```python
# autogluon/common/utils/utils.py
if version_saved != version_current:
...
if require_version_match:
raise AssertionError(
f"Predictor was created on version {version_saved} but is being loaded "
f"with version {version_current}. ..."
)
```

Since `1.5.0 != 1.6.1`, every cloud-trained predictor raises `AssertionError` on load.

To be clear: I am **not** asking for this check to be relaxed. I am aware of #5662, where that was proposed and declined, and of the maintainer position stated there — that AutoGluon makes no backwards-compatibility promise for loaded artifacts across any version difference. I accept that. I mention the check only to explain why a version-matched container is the only workable path for us: with the check working as intended, a 1.5.0 container and a 1.6.1 local install cannot be combined at all, and `require_version_match=False` is exactly the unsupported territory the maintainers warn about.

### 3. Locally resolved hyperparameters reference a model family absent from 1.5.0

Our pipeline resolves a preset to its hyperparameters dict locally (in order to cap each model's history look-back before submitting the job), then passes that dict explicitly to `fit()`. Under 1.6.1, `best_quality` resolves to:

```
['AutoETS', 'Chronos2', 'DeepAR', 'DirectTabular', 'DynamicOptimizedTheta',
'RecursiveTabular', 'SeasonalNaive', 'TemporalFusionTransformer', 'Toto2']
```

`Toto2` does not exist in 1.5.0. Comparing the two model registries confirms it:

```python
# autogluon.timeseries 1.5.0 (from the published wheel)
'TotoModel' in __all__ -> True
'Toto2Model' in __all__ -> False
'Chronos2Model' in __all__ -> True

# autogluon.timeseries 1.6.1 (local)
['ChronosModel', 'Chronos2Model', 'TotoModel', 'Toto2Model']
```

So even after switching to a preset name that both versions accept, a run that forwards locally resolved hyperparameters hits a second wall: `Chronos2` resolves fine, `Toto2` does not. Resolving presets locally and executing them remotely is only safe when both sides run the same AutoGluon version.

## Related issues

- **#2994** — the same request for the 0.7.0 image, closed with "the DLC image is already available". I could not find an equivalent open issue for 1.6.x. https://github.com/autogluon/autogluon/issues/2994
- **#5662** — proposal to relax the strict version check on load, closed as `wontfix`. Referenced above; this issue does not revisit it. https://github.com/autogluon/autogluon/issues/5662

## Questions

1. Is a 1.6.x AutoGluon training (and inference) DLC planned, and is there a rough target date?
2. Is there a rule of thumb for how long after a PyPI release the corresponding container usually appears? Knowing the typical lag would let us decide whether to pin our local environment to the container version instead of waiting.

## Current workaround

We have paused cloud training and run training locally on 1.6.1 until a matching container is available. Pinning the local installation down to 1.5.0 is the alternative we are weighing, so an indication of the expected timeline would directly inform that decision.

Start with autogluon.cloud.utils.dlc_utils, especially retrieve_available_framework_versions and retrieve_latest_framework_version, to understand how published training images are exposed. Review the DLC publishing path and the version compatibility concerns described in the issue. Done means a version-matched 1.6.x training image is published and discoverable, with inference coverage clarified.

Written by the indexing model from the issue text.