microsoft / microsoft/AzureStorageExplorer

Default Transfer settings seem buggy/poorly chosen?

Open
#3,117 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

:gear: azcopy
Dominant language
No language data
Stars
452
Forks
92
Avg merge
15h 20m
Merged PRs (30d)
3

Description

Not a bug report yet - wanted to discuss it first. Happy to raise a proper bug report if it seems relevant after a brief discussion.

If uploading GBs of data in many files, via a low-upload-bandwidth connection the default Transfer settings seem not to work? Particularly the Network and File Concurrency.

I've got mass uploads working just fine, but only because I actively set those values both to be 1.
If I leave them blank (as is their default state) AzCopy attempts to upload 10s of 8MB chunks simultaneously, which ends up meaning that 90% of the chunks timeout and the copy spends its whole handling timeouts and re-sending the same chunks over and over.
Chunks do very occasionally get through, but it's pretty rare.

Conversely, when I limit the concurrency so that it's only uploading 1 chunk from 1 file at a time, it's saturating the network connection (assuming I leave Max Transfer Rate blank) and neatly uploading every file in order.

The defaults end up with a thing that doesn't work, and setting Concurrency to "1" works perfectly and is saturating my connection. So it's not like "it works but it's less efficient". We can't up load the data any quicker than the limit of the uplink connection!


Do you agree that (if my experience is representative) this is a bug and the defaults should be changed?

If so, I'll verify that I can reproduce, adn raise an actual bug.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names AzCopy transfer settings, especially Network and File Concurrency, but provides no source files, tests, or entry points. Start by reproducing a multi-file upload with the default settings versus both concurrency values set to 1; done means confirming the behavior and identifying whether the defaults should change.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.