IntelLabs / IntelLabs/vdms

S3 Connection Layer Does Not Support Large Files

Open
#180 1 comment 0 reactions 0 assignees View on GitHub
Bug Discussion Needed Priority: Backlog
Dominant language
C++
Stars
93
Forks
34
PR merge metrics
No merged PRs in 30d

Description

**Issue:** The basic AWS S3 connection interface doesn't support file uploads greater than a certain size. I believe the hard limit is 5GB, but AWS recommends multipart upload for any files larger than 100MB. Currently, attempts to upload large files may fail, with unpredictable results (i.e. client may think the upload succeeded, but it silently fails (only prints log message)).

**Solution:** Need to implement "multipart upload" for any file larger than a certain threshold.

Contributor guide

Open the contributing guide

Research direction

Start by locating the S3 connection layer and its current file-upload path, then review how oversized uploads are handled and how failures are reported. Implement multipart upload for files above the selected threshold, and verify that successful completion and upload failures are both surfaced reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, cpp
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.