awslabs / awslabs/filemoverexpress

[Feature] Use io.ReaderAt interface for S3 uploads to improve transfer performance accuracy

Open
#4 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
4
Forks
1
Avg merge
9h 54m
Merged PRs (30d)
41

Description

## Background

The AWS SDK v2 transfer manager supports `io.ReaderAt` for multipart uploads, which allows it to read file parts concurrently at arbitrary offsets without seeking. `FileReader` already implements `ReadAt`, but `UploadConfig.Reader` is typed as `io.Reader` — so the SDK never sees the capability and the benefit is lost.

## The real problem: transfer speeds are measured at the wrong place

The comment in `file_reader.go` is telling:

// FileReader is a wrapper struct around file read operations,
// allowing us to track rudimentary read speeds from disk

`BytesRead()` counts bytes read from local disk, not bytes delivered to S3. This means the transfer speed and progress shown to users reflects how fast the file is being read off disk — not how fast it's actually being uploaded over the network. For fast local storage (NVMe, RAID), disk reads will complete far ahead of the network, making reported speeds and progress inaccurate.

The AWS SDK v2 transfer manager's progress listener support (added in [#3041](https://github.com/aws/aws-sdk-go-v2/pull/3041) and [#3083](https://github.com/aws/aws-sdk-go-v2/pull/3083)) provides a way to track bytes as they are sent over the network, which is what users actually care about.

## Proposed changes

1. Change `UploadConfig.Reader` from `io.Reader` to `io.ReaderAt` so the SDK manager can use concurrent part reads
2. Hook into the SDK transfer manager's progress listener to track bytes delivered to S3 rather than bytes read from disk
3. Update `BytesRead()` / progress reporting to reflect actual network throughput

## Why this matters for users

- Transfer speed shown in the GUI and CLI will reflect actual upload speed, not disk read speed
- Large file transfers on fast local storage will show correct ETAs

## Relevant files

- `src/cli/core/transfer-api/upload.go` — `UploadConfig.Reader` field
- `src/cli/core/transfer-api/file_reader.go` — `BytesRead()` currently measures disk reads
- `src/cli/core/job_manager/transfer-worker.go` — consumes `BytesRead()` for progress updates

Contributor guide

Open the contributing guide

Research direction

Start with src/cli/core/transfer-api/upload.go to trace UploadConfig.Reader, then inspect ReadAt and BytesRead() in src/cli/core/transfer-api/file_reader.go and their use in src/cli/core/job_manager/transfer-worker.go. The work is complete when multipart uploads use ReaderAt and progress reporting reflects bytes delivered to S3 rather than local disk reads.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.