Folders with a very large number of files are showing as empty in the iOS app

Steps to reproduce

1. Open NextCloud iOS app
2. Choose 'Files' tab
3. Navigate to an external storage folder which contains a lot of media files (in my case ~20,000 photo and video files, totalling ~130Gb)

Expected behaviour

Similar to the behaviour in the web app, after a short wait (~20 seconds), a list of the files is displayed.

Actual behaviour

After a longer wait (~60 seconds), the iOS app indicates that there are no files in the directory.

Logs

There are 2 issues at play, which combine to cause the main issue discussed here:

1. There appears to be a 60s timeout on the WebDAV call being made from the iOS app
2. The WebDav call to retrieve the contents of this folder from the iOS app is taking a much longer time to respond than the equivalent (but different) call from the web app (~300 seconds vs ~15 seconds)

The reason I suspect that the WebDAV call from within the iOS app has a 60s timeout is due to the iOS logs consistently reporting a timeout after 60s. When I monitor http traffic via a web proxy, there is no timeout shown and the request runs to completion (meaning that the timeout is client side rather than server side).

iOS Logs:

2021-09-09 02:08:41 Network request started: PROPFIND http://redacted/remote.php/webdav/redacted/info/
2021-09-09 02:09:41 Network response request: http://redacted/remote.php/webdav/redacted/info/, result: failure(Alamofire.AFError.sessionTaskFailed(error: Error Domain=NSURLErrorDomain Code=-1001 "The request timed out." UserInfo={_kCFStreamErrorCodeKey=-2102, NSUnderlyingError=0x281220780 {Error Domain=kCFErrorDomainCFNetwork Code=-1001 "(null)" UserInfo={_kCFStreamErrorCodeKey=-2102, _kCFStreamErrorDomainKey=4}}, _NSURLErrorFailingURLSessionTaskErrorKey=LocalDataTask <133556B0-57FB-41D5-9F46-8C3996F5022F>.<118>, _NSURLErrorRelatedURLSessionTaskErrorKey=(
    "LocalDataTask <133556B0-57FB-41D5-9F46-8C3996F5022F>.<118>"
), NSLocalizedDescription=The request timed out., NSErrorFailingURLStringKey=http://redacted/remote.php/webdav/redacted/info/, NSErrorFailingURLKey=http://redacted/remote.php/webdav/redacted/info/, _kCFStreamErrorDomainKey=4}))

As mentioned earlier, in order to investigate this further, I redirected both the iOS app and the Web app through the same http traffic monitoring proxy. This monitor consistently shows the WebDAV request from the iOS app taking ~300s, and the equivalent request from the Web app taking ~15s. The two requests are different, but they ultimately appear to provide very similar responses (see below).

iOS WebDAV request:

PROPFIND /remote.php/webdav/redacted/info/ HTTP/1.1
Host: redacted
OCS-APIRequest: true
Accept: */*
Authorization: Basic redacted
Accept-Encoding: br;q=1.0, gzip;q=0.9, deflate;q=0.8
Accept-Language: en;q=1.0, fr;q=0.9, de;q=0.8, ja;q=0.7, nl;q=0.6, it;q=0.5
Content-Type: application/xml
Content-Length: 1527
Depth: 1
User-Agent: Mozilla/5.0 (iOS) Nextcloud-iOS/4.0.6
Cookie: redacted
Connection: keep-alive


<?xml version="1.0" encoding="UTF-8"?>
<d:propfind xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns" xmlns:nc="http://nextcloud.org/ns">
    <d:prop>
        <d:getlastmodified />
        <d:getetag />
        <d:getcontenttype />
        <d:resourcetype />
        <d:quota-available-bytes />
        <d:quota-used-bytes />

        <permissions xmlns="http://owncloud.org/ns"/>
        <id xmlns="http://owncloud.org/ns"/>
        <fileid xmlns="http://owncloud.org/ns"/>
        <size xmlns="http://owncloud.org/ns"/>
        <favorite xmlns="http://owncloud.org/ns"/>
        <share-types xmlns="http://owncloud.org/ns"/>
        <owner-id xmlns="http://owncloud.org/ns"/>
        <owner-display-name xmlns="http://owncloud.org/ns"/>
        <comments-unread xmlns="http://owncloud.org/ns"/>
        <checksums xmlns="http://owncloud.org/ns"/>
        <downloadURL xmlns="http://owncloud.org/ns"/>
        <data-fingerprint xmlns="http://owncloud.org/ns"/>

        <creation_time xmlns="http://nextcloud.org/ns"/>
        <upload_time xmlns="http://nextcloud.org/ns"/>
        <is-encrypted xmlns="http://nextcloud.org/ns"/>
        <has-preview xmlns="http://nextcloud.org/ns"/>
        <mount-type xmlns="http://nextcloud.org/ns"/>
        <rich-workspace xmlns="http://nextcloud.org/ns"/>
        <note xmlns="http://nextcloud.org/ns"/>

        <share-permissions xmlns="http://open-collaboration-services.org/ns"/>
        
        <share-permissions xmlns="http://open-cloud-mesh.org/ns"/>
    </d:prop>
</d:propfind>

iOS WebDAV response (duration ~300s):

HTTP/1.1 207 Multi-Status
Date: Wed, 08 Sep 2021 23:55:55 GMT
Server: Apache/2.4.41 (Ubuntu)
Referrer-Policy: no-referrer
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Robots-Tag: none
X-XSS-Protection: 1; mode=block
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Content-Security-Policy: default-src 'none';
Vary: Brief,Prefer
DAV: 1, 3, extended-mkcol
Keep-Alive: timeout=5, max=99
Transfer-Encoding: chunked
Content-Type: application/xml; charset=utf-8
Proxy-Connection: keep-alive


<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:oc="http://owncloud.org/ns" xmlns:nc="http://nextcloud.org/ns">
...
</d:response>
<d:response>
    <d:href>/remote.php/webdav/redacted/info/09D71C68-5CCA-469F-AAA1-C23F2AC7DC0F.JPG</d:href>
    <d:propstat>
        <d:prop>
            <d:getlastmodified>Sun, 09 Dec 2018 19:11:41 GMT</d:getlastmodified>
            <d:getetag>&quot;9f46f89482e0a3f145f0226011a52747&quot;</d:getetag>
            <d:getcontenttype>image/jpeg</d:getcontenttype><d:resourcetype/>
            <oc:permissions>RMGDNVW</oc:permissions>
            <oc:id>00016539ocum9e0hf7du</oc:id>
            <oc:fileid>16539</oc:fileid>
            <oc:size>1198042</oc:size>
            <oc:favorite>0</oc:favorite>
            <oc:share-types/>
            <oc:owner-id>redacted</oc:owner-id>
            <oc:owner-display-name>redacted</oc:owner-display-name>
            <oc:comments-unread>0</oc:comments-unread>
            <oc:downloadURL></oc:downloadURL>
            <oc:data-fingerprint></oc:data-fingerprint>
            <nc:creation_time>0</nc:creation_time>
            <nc:upload_time>0</nc:upload_time>
            <nc:has-preview>true</nc:has-preview>
            <nc:mount-type>external</nc:mount-type>
            <nc:note></nc:note>
            <x1:share-permissions xmlns:x1="http://open-collaboration-services.org/ns">19</x1:share-permissions>
            <x2:share-permissions xmlns:x2="http://open-cloud-mesh.org/ns">[&quot;share&quot;,&quot;read&quot;,&quot;write&quot;]</x2:share-permissions>
        </d:prop>
        <d:status>HTTP/1.1 200 OK</d:status>
    </d:propstat>
    <d:propstat>
        <d:prop>
            <d:quota-available-bytes/>
            <d:quota-used-bytes/>
            <oc:checksums/>
            <nc:is-encrypted/>
            <nc:rich-workspace/>
        </d:prop>
        <d:status>HTTP/1.1 404 Not Found</d:status>
    </d:propstat>
</d:response>
<d:response>
...
</d:multistatus>

Web app WebDAV request:

PROPFIND /nextcloud/remote.php/dav/files/user/redacted/info HTTP/1.1
Host: redacted
Accept: */*
X-Requested-With: XMLHttpRequest
Accept-Encoding: gzip, deflate
Accept-Language: en-gb
Content-Type: application/xml; charset=utf-8
Origin: redacted
Depth: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15
Content-Length: 653
Cookie: redacted
requesttoken: redacted
Connection: keep-alive


<?xml version="1.0"?>
<d:propfind  xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns" xmlns:nc="http://nextcloud.org/ns" xmlns:ocs="http://open-collaboration-services.org/ns">
  <d:prop>
    <d:getlastmodified />
    <d:getetag />
    <d:getcontenttype />
    <d:resourcetype />
    <oc:fileid />
    <oc:permissions />
    <oc:size />
    <d:getcontentlength />
    <d:quota-available-bytes />
    <nc:has-preview />
    <nc:mount-type />
    <nc:is-encrypted />
    <ocs:share-permissions />
    <oc:tags />
    <oc:favorite />
    <oc:comments-unread />
    <oc:owner-id />
    <oc:owner-display-name />
    <oc:share-types />
  </d:prop>
</d:propfind>

Web app WebDAV response (duration ~15s):

HTTP/1.1 207 Multi-Status
Date: Wed, 08 Sep 2021 23:55:09 GMT
Server: Apache/2.4.41 (Ubuntu)
Referrer-Policy: no-referrer
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Robots-Tag: none
X-XSS-Protection: 1; mode=block
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Content-Security-Policy: default-src 'none';
Vary: Brief,Prefer
DAV: 1, 3, extended-mkcol, access-control, calendarserver-principal-property-search, nc-calendar-search, nc-enable-birthday-calendar
Content-Encoding: gzip
Keep-Alive: timeout=5, max=95
Transfer-Encoding: chunked
Content-Type: application/xml; charset=utf-8
Proxy-Connection: keep-alive


<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:oc="http://owncloud.org/ns" xmlns:nc="http://nextcloud.org/ns">
...
</d:response>
<d:response>
    <d:href>/nextcloud/remote.php/dav/files/user/redacted/info/09D71C68-5CCA-469F-AAA1-C23F2AC7DC0F.JPG</d:href>
    <d:propstat>
        <d:prop>
            <d:getlastmodified>Sun, 09 Dec 2018 19:11:41 GMT</d:getlastmodified>
            <d:getetag>&quot;9f46f89482e0a3f145f0226011a52747&quot;</d:getetag>
            <d:getcontenttype>image/jpeg</d:getcontenttype><d:resourcetype/>
            <oc:fileid>16539</oc:fileid>
            <oc:permissions>RMGDNVW</oc:permissions>
            <oc:size>1198042</oc:size>
            <d:getcontentlength>1198042</d:getcontentlength>
            <nc:has-preview>true</nc:has-preview>
            <nc:mount-type>external</nc:mount-type>
            <x1:share-permissions xmlns:x1="http://open-collaboration-services.org/ns">19</x1:share-permissions>
            <oc:tags/>
            <oc:favorite>0</oc:favorite>
            <oc:comments-unread>0</oc:comments-unread>
            <oc:owner-id>redacted</oc:owner-id>
            <oc:owner-display-name>redacted</oc:owner-display-name>
            <oc:share-types/>
        </d:prop>
        <d:status>HTTP/1.1 200 OK</d:status>
    </d:propstat>
    <d:propstat>
        <d:prop>
            <d:quota-available-bytes/>
            <nc:is-encrypted/>
        </d:prop>
        <d:status>HTTP/1.1 404 Not Found</d:status>
    </d:propstat>
</d:response>
<d:response>
...
</d:multistatus>

Proposed Solution

There are two potential solutions for the two issues. Preferably both would be implemented, but implementing only one should fix the main issue described:

1. Extend the iOS client timeout for WebDAV requests to allow for very long response times, which would cater to handling very large folders.
2. Change the iOS app's WebDAV request for folders, in to bring it into line with the NextCloud web app, which would reduce the overall response time for this type of request significantly.

Reasoning or why should it be changed/implemented?

Currently folders containing a very large number of files are displayed as empty in the iOS app. This incorrect view of a folder could create many undesirable outcomes (e.g. deleting incredibly large, populated folders, because the user believes they are empty due to the app reporting this state). It is also a huge inconvenience if you can't access files which you safely stored in NextCloud.

Environment data

iOS version: iOS 14.7.1

Nextcloud iOS app version: 4.0.6

Server operating system: Ubuntu 20.04.3 LTS

Web server: Apache 2

Database: MySQL 8

PHP version: PHP 7.4

Nextcloud version: Nextcloud 22.1.1

Start with the iOS WebDAV folder request and its Alamofire timeout behavior, then compare the iOS PROPFIND properties with the web app request. Done means a very large external-storage folder is not shown as empty and remains usable when the response takes longer than 60 seconds.