Feature request: local storage (also a prediction)

This is both a feature request as well as a prediction.

(See also: Issue #3179, probably other similar requests in comments attached to other issues.)

Currently file synchronization programs take three different approaches, as observed by me in a Linux environment. (See note about Android below.)

A. Store all files locally, detect updates and synchronize. E.g., Dropbox.

B. Store all files in a virtual filesystem mounted to look local. Updates are implemented in the filesystem itself so no separate synchronization mechanism is needed. E.g., Keybase.

C. Store all files in a virtual filesystem as in B above, but allow the user to allocate local storage outside the virtual filesystem for selected subtrees, which are then synchronized via a separate mechanism. E.g., pCloud.

(Conflict detection and resolution will always be needed, since operations over the Internet will not be atomic.)

The big advantage of A and the local storage part of C is that a synchronized directory remains fully available regardless of problems in upstream connectivity. So one can use a synchronized directory as a working area, in which files can be actually edited and software compiled, with changes being reflected in any remote synced copies on other machines. This methodology works even in the face of network problems -- files will get synced as soon as connectivity is restored.

I think it is because of the advantages of A in the face of network issues that Dropbox is so immensely popular. (That, plus its excellent near-real-time synchronization, plus relatively good multi-platform support.)

The big advantages of B and C are that the virtual filesystem can be quite large while taking up very little local storage. So the user could be offered multiple terabytes of storage even on a local machine that has much less. This can be a big win on mobile devices.

C gives us the best of both worlds. Mapping one or more subdirectory trees to local storage allows us to use that as an active work area despite network problems. The rest of the virtual filesystem gives us the benefit of large amounts of storage that takes up no space on our local device.

(Generous amounts of caching can make B act almost like A or C, but the software will sometimes guess incorrectly, and a network problem will leave the user unable to access needed files.)

Now let me explain why either A or C is a business necessity, and no software that provides only B will ever be successful in the market. As open source for a few of us, yes. As a competitor in the market, no.

The reason is that users have been spoiled by Dropbox, and have come to expect that their synchronized directories will respond instantly and be always available (locally) regardless of network issues. I've even come across postings on the Internet describing how a Dropbox folder is used for editing files on a local machine while being concurrently used to compile the same software in a synced directory on a remote server.

These spoiled users will never want to rely solely on any software that only provides a network-dependent filesystem. So no matter how good Keybase ever becomes, it will never replace Dropbox-like competitors. And if you don't replace them, then they get most of the business.

The counterargument to this is that Keybase provides security that Dropbox and its direct competitors don't. But some of the newer competitors like pCloud and sync.com do offer client-side encryption. So those offerings that offer both client-side encryption and local storage will eventually prevail.

**Feature request:** Keybase should provide an option for local storage. Syncing should be fast and reliable. Conflict management should be easy and as transparent as possible.

**Prediction:** Keybase will end up providing local storage, because otherwise, I don't think the market will treat it kindly.

Note about Android. It's not clear to me exactly how syncing programs implement virtual filesystems in an Android environment. pCloud appears to create a virtual filesystem but it is not mounted in the normal sense of the word, and there is no way to get a path to a file in this virtual filesystem that looks like a normal Linux path. I think Keybase might do something similar. The result is that you cannot simply point an arbitrary app towards a file in one of these virtual filesystems, which means they are difficult or annoying to use.

Edited to fix typos.

The issue names no files, tests, or entry points; start by reviewing related issue #3179 and the current filesystem and synchronization architecture. Done would require a defined local-storage design with reliable syncing and transparent conflict management, but the issue does not specify implementation boundaries or acceptance tests.

Written by the indexing model from the issue text.