awslabs / awslabs/filemoverexpress

Glacier objects fail on download with no reason shown in the job's Logs tab

Open
#95 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
TypeScript
Stars
4
Forks
1
Avg merge
9h 54m
Merged PRs (30d)
41

Description

## Summary

When downloading an S3 prefix that contains objects in the **Glacier Flexible Retrieval** (`GLACIER`) storage class, those files fail with a generic `Error` status, and **the job's Logs tab shows no explanation** ("No log entries for this job yet"). The user has no way to understand *why* the transfer failed from the UI.

There are two related defects behind this.

## Defect 1 — Discovery's Glacier skip misses plain `GLACIER`

`src/cli/core/discovery/s3_discovery/s3_discovery.go` skips archived objects during discovery, but only checks for Glacier Instant Retrieval and Deep Archive:

```go
// Ignore the Glacier storage class since the tool doesn't support it
if s3Object.StorageClass == string(types.ObjectStorageClassGlacierIr) ||
s3Object.StorageClass == string(types.ObjectStorageClassDeepArchive) {
err = fmt.Errorf(discovery.StrDownloadGlacierFileError, s3Object.Key, s3Object.StorageClass)
discoveryErrors = append(discoveryErrors, err)
continue
}
```

It does **not** check plain `GLACIER` (Glacier Flexible Retrieval, `types.ObjectStorageClassGlacier`). As a result, a plain-`GLACIER` object passes the filter, becomes a real download task, and only fails later at `GetObject` time with S3's `InvalidObjectState` error ("The operation is not valid for the object's storage class"). Objects in restore-required classes should be caught up front in discovery like the other two.

## Defect 2 — The failure reason is never surfaced in the job's Logs tab

This is the more user-visible problem.

When a download task fails, `src/cli/core/job_manager/transfer-worker.go` attaches the real error to the task (`task.SetStatusAndError(TaskStatusError, err)` — this is what the "Copy error" button retrieves) and also emits it:

```go
events.Events.Error(
"Failed to transfer file %s: %s",
filePath,
err.Error(),
)
```

But `EventBus.Error()` (`src/cli/events/bus.go`) produces a generic `MessageEvent` with **no job ID**:

```go
evt := &eventtypes.MessageEvent{Msg: logger.FormatLogMessage(message, args), EventPriority: logger.ErrorLevel}
eb.Send(evt)
```

On the GUI side, `LogsService.handleMessage()` (`src/gui/src/app/services/logs/logs.service.ts`) stores that entry with `jobId: null`:

```ts
private handleMessage(evt: CoreEvents.MessageEvent) {
this.store.dispatch(addLog({
log: { level: evt.logLevel, message: evt.msg, timestamp: new Date(), jobId: null },
}));
}
```

The Job Details **Logs** tab only renders log entries whose `jobId` matches the current job. Because the per-file failure message carries `jobId: null`, it is filtered out — so the tab shows "No log entries for this job yet" even when files have failed.

The Glacier-skip warning that discovery *does* emit (`events.Events.Warn`, for GLACIER_IR / DEEP_ARCHIVE) has the same problem: it is a job-less `MessageEvent`, so it is also invisible in the job-scoped log.

## Steps to reproduce

1. Have an S3 prefix that contains one or more objects in the `GLACIER` (Flexible Retrieval) storage class.
2. Download that prefix with FME.
3. Observe: the Glacier object(s) fail with a red `Error` status and 0% progress.
4. Open the failed job → **Logs** tab.

## Expected

- Objects in restore-required storage classes (`GLACIER`, `GLACIER_IR`, `DEEP_ARCHIVE`) are detected during discovery and either skipped with a clear reason or reported as a specific, human-readable failure.
- The job's **Logs** tab shows the reason a file failed (e.g. "skipped/failed: object is in GLACIER storage class and must be restored before download"), scoped to that job.

## Actual

- Plain `GLACIER` objects are not caught in discovery; they fail late with a raw `InvalidObjectState` error.
- The job's **Logs** tab is empty ("No log entries for this job yet") because the failure event is emitted without a job ID and is filtered out of the job-scoped view. The reason is only reachable via the "Copy error" button on the individual file.

## Suggested fix

1. Add `types.ObjectStorageClassGlacier` to the storage-class check in `s3_discovery.go` (and consider centralizing the "restore-required" set so discovery, rename, and download all agree).
2. Give per-task transfer failures a job-scoped event (carry the job ID) so they land in the job's Logs tab — either add a job ID to the error event path used by the transfer worker, or route task failures through an existing job-scoped event type. Consider mapping `InvalidObjectState` to a clear, actionable message.

## Environment

- Component: Angular GUI + Go daemon
- Direction: S3 → local (download)

Contributor guide

Open the contributing guide

Research direction

Start with src/cli/core/discovery/s3_discovery/s3_discovery.go, src/cli/core/job_manager/transfer-worker.go, src/cli/events/bus.go, and src/gui/src/app/services/logs/logs.service.ts; trace how discovery and transfer errors reach the job-scoped Logs tab. Done means restore-required S3 classes are handled during discovery and the relevant failure reason appears in the correct job's Logs tab.

Written by the indexing model from the issue text.

Assessment

Tech stack
angular, aws, go, typescript
Domain
backend, cloud, frontend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.