awslabs / awslabs/filemoverexpress

Uploads can silently drop files at high maxActiveTransfers (job reports success)

Open
#111 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
4
Forks
1
Avg merge
9h 54m
Merged PRs (30d)
41

Description

## Summary
With a high `general.maxActiveTransfers` (MAT), an upload job can finish with **fewer objects in the destination than in the source while the job still completes without surfacing an error**. This is a silent partial-upload / data-integrity issue: the user believes the upload succeeded, but files are missing.

## Observed
Benchmarking FME CLI uploads (auto-tuning on, checksums off) of a 2,400-file dataset (20 MB each, served from RAM) to S3 in us-west-2, sweeping MAT:

| maxActiveTransfers | objects in destination | result |
|---|---|---|
| 100 | 2400 / 2400 | OK |
| 200 | 2400 / 2400 | OK |
| 256 | 2400 / 2400 | OK |
| **512** | **2256 / 2400 (144 missing)** | **no error surfaced** |

At MAT 512 throughput also regressed below the MAT 256 result, consistent with resource exhaustion rather than a transient network blip.

## Impact
Silent partial upload. Because no error is surfaced, a user — or an automated pipeline / hot folder — would not know 144 files never landed. For media delivery/archival this is a data-integrity risk.

## Likely cause (to confirm)
FME's concurrent S3 connections scale roughly as `MAT × per-file multipart threads`. Auto-tuning uses ~10 threads for ~20 MB files, so MAT 512 implies ~5,000 concurrent connections/file descriptors, which can exceed the process open-file limit (`ulimit -n`). Uploads that fail to acquire an fd/connection appear to be dropped without failing the job. Cause not yet confirmed from logs — but the core defect is that, whatever the underlying failure, it is not surfaced and the job is not marked failed/partial.

## Suggested fixes
1. **Never report success when tasks failed or objects are missing** — surface per-file errors and mark the job failed/partial.
2. **Bound effective concurrency by the open-file limit** — cap `MAT × threads` under `ulimit -n` (with margin) and/or raise the soft fd limit at startup.
3. Consider **back-pressure** so excess concurrency queues rather than silently fails.

## Related
- Ties into the `maxActiveTransfers` default / auto-scaling discussion (a higher default helps throughput but must be fd-safe).
- Many-file throughput fix: #110 (parallel object-already-exists filter + MAT default 10 -> 100).

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported S3 upload benchmark while sweeping maxActiveTransfers, especially 512, and inspect logs or failure handling when uploads cannot acquire resources. Trace how per-file transfer failures affect the job result. Done means missing or failed uploads are surfaced and the job cannot report success for a partial upload.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.