Improve support for transferring large files when operating on small nodes connected to external stores

Background

The basic file storage mechanism employs the local file system, but deployments are increasingly relying on external storage systems such as S3, due to elasticity and other advantages.

In cloud deployments, S3 generally allows storage capacity scaling to arbitrary sizes within practical limits, without requiring manual maintenance such as resizing a file system. It also easily accommodates billing agreements based on prorated usage.

For deployments relying heavily on external storage, requirements for memory and disk may be, at least in principle, quite minimal for a node hosting the application, especially if frequent multi-user access is not required, or if the application primarily serves file transfers and basic applications, such as contacts and calendar.

However, the current application design may not support optimal utilization of available local hardware resources, raising minimum requirements above those otherwise needed.

Problem

In spite of the appeal for deployments relying on external storage, the application imposes considerable requirements for systems to support transfer of large files. Use of such resources is temporary within the duration of a transfer operation.

It is plain that the current application design expects that deployments managing large files are not operating on small nodes with limited local resources.

Such expectations may lead to much higher operating costs than strictly needed for any reason except software limitations.

Especially if the node operating the application is connected by a high-speed link to external storage, as would be common for cloud deployments, the practical benefit is quite limited to data temporarily being stored on a local node.

Request

Requested is stronger support for the application operating on smaller nodes, while still processing transfers of large files stored in S3.

Ideally, the application naturally would support transferring files of arbitrary size without requiring available memory or disk far exceeding that required for other use of the application.

Optimal interaction with large stores would allow customers to invest in storage solutions that are economical at scale, instead of investing unnecessarily in nodes hosting applications.

Remarks

The scope of the topic is intended to be for discussion, hopefully evolving into determining a set of specific modifications to the software, in order to fulfill the purpose of improving customer experience within certain scenarios.

If useful, the topic may then be used to track various concrete modifications, or a new one may opened for the purpose.

No files, tests, or entry points are named. Start by mapping the large-file transfer and external-storage paths, especially how S3-backed files use local memory and disk. The issue needs to become concrete before implementation; done should mean large transfers work with substantially reduced local resource requirements.