owncloud / owncloud/ocis

Resumable uploads should not affect existing files until postprocessing has finished

Open
#7,246 0 comments 0 reactions 0 assignees View on GitHub
Type:Bug
Dominant language
Go
Stars
2.1k
Forks
274
Avg merge
2d 1h
Merged PRs (30d)
103

Description

Today, under the shower, I realized that tus, resumable uploads etc are just a way to prevent corrupted data. Comparable to writing a temp file and then using move to make it an 'atomic' write. Just as in POSIX world you can directly write to a file or use a more robust 'write and move tmp file'. In HTML / WabDAV that would be PUT or POST/tus upload.

BUT

This has interesting implications. Writing a file directly / PUT is/should always be synchronous. The fclose call is intended to let the storage do everything that is necessary so that the next read can immediately read the new data. When writing a temp file POSIX clients are absolutely aware that they are not directly writing to the destination file. In oc10 we are also writing .part files to prevent corrupting any existing file. So PUT should always be sync and will have to wait for any postprocessing.

Using Tus is a safe way for clients to write large files without having to wait for server processing. The tus upload however is the equivalent of a temp file. Clients that use it to upload files MUST keep track of the uploads they started. Currently @hvonreth explained that the desktop eg. will blindly compare the etag (and content hash) of a directory listing / PROPFIND with its local database and download old data.

In order to indicate that a file is about to change the graph api driveItems have a pendingOperations property. For WebDAV we could add another property *shudder* but I would prefer finally implementing `/drive/{driveid}/item/{itemid}/` and `/drive/{driveid}/item/{itemid}/children`. In any case this is only an indicator for other clients. The client that started an upload needs to keep track of it. We can even notify clients of finished uploads with SSE.

As a data platform we need to make sure we handle the direct file writes in a POSIX compatible way. For a FUSE filesystem that means: file writes must be atomic and synchronous. Furthermore, we need to detect 'write + move tmp file' to keep the file id. In POSIX land we cannot just accept a file close and return IO errors until postprocessing has finished.

ASYNC vs SYNC file processing as not a server config option: clients can decide how they want to upload files - either us a synchronous PUT, which might work for small files or use POST to write a temp file that the server will move automatically (this is a difference to real temp files which the client moves).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.