Helix service truncates DockerTag which causes "invalid reference format" errors when a job is retried
- Dominant language
- C#
- Stars
- 31
- Forks
- 38
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 24
Description
Runtime has jobs that submit using a sha256 docker tag like this: `mcr.microsoft.com/dotnet-buildtools/prereqs:windowsservercore-ltsc2025-helix-webassembly-amd64@sha256:0f6feb7857a2908a440fb4df0f06f19322dc356ddcc4710d2195e9b8133a9420`
When a work item fails and is retried by the Job Monitor, the subsequent work item fails with:
```
2026-09-01T18:52:06.324Z WARNING dockerhelper(160) get_docker_image_info Unable to fetch manifest for mcr.microsoft.com/dotnet-buildtools/prereqs:windowsservercore-ltsc2025-helix-webassembly-amd64@sha256:0f6feb7857a2908a440fb4d...: invalid reference format
2026-09-01T18:52:06.325Z WARNING dockerhelper(264) pull Could not fetch image info for mcr.microsoft.com/dotnet-buildtools/prereqs:windowsservercore-ltsc2025-helix-webassembly-amd64@sha256:0f6feb7857a2908a440fb4d..., but proceeding with pull
2026-09-01T18:52:06.326Z INFO dockerhelper(267) pull Starting pull operation for mcr.microsoft.com/dotnet-buildtools/prereqs:windowsservercore-ltsc2025-helix-webassembly-amd64@sha256:0f6feb7857a2908a440fb4d......
2026-09-01T18:52:06.326Z INFO dockerhelper(288) pull Pull attempt 1/5 for mcr.microsoft.com/dotnet-buildtools/prereqs:windowsservercore-ltsc2025-helix-webassembly-amd64@sha256:0f6feb7857a2908a440fb4d...
2026-09-01T18:52:06.377Z WARNING dockerhelper(339) pull Pull attempt 1 failed after 0.05 seconds: docker pull exited with code 1: invalid reference format
2026-09-01T18:52:06.377Z INFO dockerhelper(120) log_disk_snapshot Disk snapshot (retry-1) on C:\: 49.83/126.51 GB used (39.39%), 76.68 GB free
```
This is because the image tag got truncated (the `...` is NOT just abbreviated in the log, it is actually the real string).
Copilot analysis:
----
The SQL processor explicitly truncates `DockerTag` to 128 characters:
[src/ServiceFabric/Helix/SqlProcessor/SqlProcessorEventBatchHandler.cs:206](https://dev.azure.com/dnceng/internal/_git/dotnet-helix-service?path=/src/ServiceFabric/Helix/SqlProcessor/SqlProcessorEventBatchHandler.cs&line=206)
```csharp
jobData.Data["DockerTag"] =
SqlHelper.TruncateString(
currentMessage.Value("DockerTag")?.ToLowerInvariant(),
128);
```
The database column has the corresponding limit:
[src/helixdata/Tables/Jobs.sql:27](https://dev.azure.com/dnceng/internal/_git/dotnet-helix-service?path=/src/helixdata/Tables/Jobs.sql&line=27)
```sql
[DockerTag] VARCHAR (128) NULL,
```
`SqlHelper.TruncateString()` preserves the first `size - 3` characters and appends `...`:
```csharp
return value.Substring(0, size - 3) + "...";
```
The complete failure chain is:
1. Runtime initially submits the full 166-character Docker reference.
2. The original Helix controller message still contains the full value, so the first job runs successfully inside the container for about 37 minutes before the test itself exits with code `1`.
3. The `JobStarted` event is persisted to SQL, where `DockerTag` becomes `first 125 characters + "..."`.
4. Arcade JobMonitor retries the failed work item. Its resubmission code reads `Job.DetailsAsync()` and copies the SQL-backed value:
```csharp
DockerTag = details.DockerTag,
```
5. Retry jobs `15105efe-...` and `0d17607d-...` therefore receive the literal truncated Docker reference and fail during pull with exit code `-4`.
So this is specifically an interaction between the **Helix SQL `VARCHAR(128)` limit** and **Arcade JobMonitor resubmission using `JobDetails.DockerTag`**. The proper service fix is to enlarge the `Jobs.DockerTag` column and update the `TruncateString(..., 128)` limit consistently.
### Release Note Category
- [ ] Feature changes/additions
- [ ] Bug fixes
- [ ] Internal Infrastructure Improvements
### Release Note Description
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.