awslabs / awslabs/filemoverexpress
Glacier objects fail on download with no reason shown in the job's Logs tab
- Dominant language
- TypeScript
- Stars
- 4
- Forks
- 1
- Avg merge
- 9h 54m
- Merged PRs (30d)
- 41
Description
## Summary
When downloading an S3 prefix that contains objects in the **Glacier Flexible Retrieval** (`GLACIER`) storage class, those files fail with a generic `Error` status, and **the job's Logs tab shows no explanation** ("No log entries for this job yet"). The user has no way to understand *why* the transfer failed from the UI.
There are two related defects behind this.
## Defect 1 — Discovery's Glacier skip misses plain `GLACIER`
`src/cli/core/discovery/s3_discovery/s3_discovery.go` skips archived objects during discovery, but only checks for Glacier Instant Retrieval and Deep Archive:
```go
// Ignore the Glacier storage class since the tool doesn't support it
if s3Object.StorageClass == string(types.ObjectStorageClassGlacierIr) ||
s3Object.StorageClass == string(types.ObjectStorageClassDeepArchive) {
err = fmt.Errorf(discovery.StrDownloadGlacierFileError, s3Object.Key, s3Object.StorageClass)
discoveryErrors = append(discoveryErrors, err)
continue
}
```
It does **not** check plain `GLACIER` (Glacier Flexible Retrieval, `types.ObjectStorageClassGlacier`). As a result, a plain-`GLACIER` object passes the filter, becomes a real download task, and only fails later at `GetObject` time with S3's `InvalidObjectState` error ("The operation is not valid for the object's storage class"). Objects in restore-required classes should be caught up front in discovery like the other two.
## Defect 2 — The failure reason is never surfaced in the job's Logs tab
This is the more user-visible problem.
When a download task fails, `src/cli/core/job_manager/transfer-worker.go` attaches the real error to the task (`task.SetStatusAndError(TaskStatusError, err)` — this is what the "Copy error" button retrieves) and also emits it:
```go
events.Events.Error(
"Failed to transfer file %s: %s",
filePath,
err.Error(),
)
```
But `EventBus.Error()` (`src/cli/events/bus.go`) produces a generic `MessageEvent` with **no job ID**:
```go
evt := &eventtypes.MessageEvent{Msg: logger.FormatLogMessage(message, args), EventPriority: logger.ErrorLevel}
eb.Send(evt)
```
On the GUI side, `LogsService.handleMessage()` (`src/gui/src/app/services/logs/logs.service.ts`) stores that entry with `jobId: null`:
```ts
private handleMessage(evt: CoreEvents.MessageEvent) {
this.store.dispatch(addLog({
log: { level: evt.logLevel, message: evt.msg, timestamp: new Date(), jobId: null },
}));
}
```
The Job Details **Logs** tab only renders log entries whose `jobId` matches the current job. Because the per-file failure message carries `jobId: null`, it is filtered out — so the tab shows "No log entries for this job yet" even when files have failed.
The Glacier-skip warning that discovery *does* emit (`events.Events.Warn`, for GLACIER_IR / DEEP_ARCHIVE) has the same problem: it is a job-less `MessageEvent`, so it is also invisible in the job-scoped log.
## Steps to reproduce
1. Have an S3 prefix that contains one or more objects in the `GLACIER` (Flexible Retrieval) storage class.
2. Download that prefix with FME.
3. Observe: the Glacier object(s) fail with a red `Error` status and 0% progress.
4. Open the failed job → **Logs** tab.
## Expected
- Objects in restore-required storage classes (`GLACIER`, `GLACIER_IR`, `DEEP_ARCHIVE`) are detected during discovery and either skipped with a clear reason or reported as a specific, human-readable failure.
- The job's **Logs** tab shows the reason a file failed (e.g. "skipped/failed: object is in GLACIER storage class and must be restored before download"), scoped to that job.
## Actual
- Plain `GLACIER` objects are not caught in discovery; they fail late with a raw `InvalidObjectState` error.
- The job's **Logs** tab is empty ("No log entries for this job yet") because the failure event is emitted without a job ID and is filtered out of the job-scoped view. The reason is only reachable via the "Copy error" button on the individual file.
## Suggested fix
1. Add `types.ObjectStorageClassGlacier` to the storage-class check in `s3_discovery.go` (and consider centralizing the "restore-required" set so discovery, rename, and download all agree).
2. Give per-task transfer failures a job-scoped event (carry the job ID) so they land in the job's Logs tab — either add a job ID to the error event path used by the transfer worker, or route task failures through an existing job-scoped event type. Consider mapping `InvalidObjectState` to a clear, actionable message.
## Environment
- Component: Angular GUI + Go daemon
- Direction: S3 → local (download)
Contributor guide
Research direction
Start with src/cli/core/discovery/s3_discovery/s3_discovery.go, src/cli/core/job_manager/transfer-worker.go, src/cli/events/bus.go, and src/gui/src/app/services/logs/logs.service.ts; trace how discovery and transfer errors reach the job-scoped Logs tab. Done means restore-required S3 classes are handled during discovery and the relevant failure reason appears in the correct job's Logs tab.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- angular, aws, go, typescript
- Domain
- backend, cloud, frontend, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100