awslabs / awslabs/filemoverexpress
[Feature] Use io.ReaderAt interface for S3 uploads to improve transfer performance accuracy
- Dominant language
- TypeScript
- Stars
- 4
- Forks
- 1
- Avg merge
- 9h 54m
- Merged PRs (30d)
- 41
Description
## Background
The AWS SDK v2 transfer manager supports `io.ReaderAt` for multipart uploads, which allows it to read file parts concurrently at arbitrary offsets without seeking. `FileReader` already implements `ReadAt`, but `UploadConfig.Reader` is typed as `io.Reader` — so the SDK never sees the capability and the benefit is lost.
## The real problem: transfer speeds are measured at the wrong place
The comment in `file_reader.go` is telling:
// FileReader is a wrapper struct around file read operations,
// allowing us to track rudimentary read speeds from disk
`BytesRead()` counts bytes read from local disk, not bytes delivered to S3. This means the transfer speed and progress shown to users reflects how fast the file is being read off disk — not how fast it's actually being uploaded over the network. For fast local storage (NVMe, RAID), disk reads will complete far ahead of the network, making reported speeds and progress inaccurate.
The AWS SDK v2 transfer manager's progress listener support (added in [#3041](https://github.com/aws/aws-sdk-go-v2/pull/3041) and [#3083](https://github.com/aws/aws-sdk-go-v2/pull/3083)) provides a way to track bytes as they are sent over the network, which is what users actually care about.
## Proposed changes
1. Change `UploadConfig.Reader` from `io.Reader` to `io.ReaderAt` so the SDK manager can use concurrent part reads
2. Hook into the SDK transfer manager's progress listener to track bytes delivered to S3 rather than bytes read from disk
3. Update `BytesRead()` / progress reporting to reflect actual network throughput
## Why this matters for users
- Transfer speed shown in the GUI and CLI will reflect actual upload speed, not disk read speed
- Large file transfers on fast local storage will show correct ETAs
## Relevant files
- `src/cli/core/transfer-api/upload.go` — `UploadConfig.Reader` field
- `src/cli/core/transfer-api/file_reader.go` — `BytesRead()` currently measures disk reads
- `src/cli/core/job_manager/transfer-worker.go` — consumes `BytesRead()` for progress updates
Contributor guide
Research direction
Start with src/cli/core/transfer-api/upload.go to trace UploadConfig.Reader, then inspect ReadAt and BytesRead() in src/cli/core/transfer-api/file_reader.go and their use in src/cli/core/job_manager/transfer-worker.go. The work is complete when multipart uploads use ReaderAt and progress reporting reflects bytes delivered to S3 rather than local disk reads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100