Imap account with thousands of folders/mails not shown
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 1k
- Forks
- 348
- Avg merge
- 12h 28m
- Merged PRs (30d)
- 91
Description
Steps to reproduce
- Set up the imap account, save it, tested the credentials succeed.
- Starting mail app just waiting for a very very very very long time (10+ minutes), nothing appears, even folders, nothing.
- horde_imap.log shows successfully completed tasks, up to bye bye.
Expected behavior
After setup the account, mail app just start and show some emails, as the same application in the same server with different imap account with a few emails just work.
Actual behavior
- horde_imap.log filled with successfully completed tasks, like this:
Monitoring the horde_imap.log shows commands running as expected:
C: 559 UID FETCH 7605 (BODY.PEEK[2] BODY.PEEK[1] BODY.PEEK[2.MIME] BODY.PEEK[1.MIME])
S: * 555 FETCH (UID 7605 BODY[2] {26319}
S: [LITERAL DATA: 26319 bytes]
S: BODY[1] {1571}
S: [LITERAL DATA: 1571 bytes]
S: BODY[2.MIME] {80}
S: [LITERAL DATA: 80 bytes]
S: BODY[1.MIME] {96}
S: [LITERAL DATA: 96 bytes]
S: )
S: 559 OK Completed (0.001 sec)
Command 559 took 0.0841 seconds.
C: 1891 STATUS user/xxxx/yyyy/Archives/2020 (MESSAGES RECENT UIDNEXT UIDVALIDITY UNSEEN)
S: * STATUS user/xxxx/yyyy/Archives/2020 (MESSAGES 0 RECENT 0 UIDNEXT 1 UIDVALIDITY 1668461269 UNSEEN 0)
S: 1891 OK Completed
2. Task number just goes up to 14000+.
3. The app has no any output (devtools console, nextcloud.log, horde_imap.log) which could show any error.
4. The screen usually was like this (except the 42, which only appeared after I changed the imap account from user1 -> user2 with a few emails only, which was working -> change back to user1 -> still seems not working.

5. based on the imap server log and horde_imap.log (and horde_smtp.log and horde_sieve.log) the credentials are ok, mail app fetched the valid sieve script, connected successfully to the smtps and successfully fetched huge number of emails.
### Mail app version
2.2.3
### Mailserver or service
Cyrus imap 3.6.0
### Operating system
Linux Debian
### PHP engine version
PHP 8.1
### Web server
Apache (supported)
### Database
MariaDB
### Additional info
* user1 has thousands of folders and emails and access to shared mailboxes
* command line sync also happened, nothing changed
`/usr/bin/php8.1 --define apc.enable_cli=1 -f ./occ mail:account:sync --force -vvv 9` it took lot of time (10+ minutes), seems the horde_imap.log filled as expected with lot of completed tasks + bye bye included
* snappymail also behave strange, technically the same symptoms
* cannot find any useful info in php*-fpm.log, apache log, nextcloud.log
* user's filecache is empty (under data/user/cache)
* with option "show only subscribed folders" enabled, I got less sync tasks (8500+) comparing to full (14000+)
* every page reload takes 10+ minutes syncing, even when it work this could makes mail useless
* I cannot effectively debug this behaviour (as I do not see any error message, oom kill, timeout, etc., but successfully completed tasks in the horde_imap.log)
Logs:
Connection starts with this:
Wed, 15 Feb 2023 11:40:02 +0000
Connection to: imap://imap.target.tld:993/
Server connection took 0.3328 seconds.
S: * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE AUTH=PLAIN SASL-IR] cyrus36 Cyrus IMAP 3.6.0-Debian-3.6.0-1+b1 server ready
C: 1 AUTHENTICATE PLAIN [INITIAL CLIENT RESPONSE (username: user1)]
S: 1 OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL ANNOTATE-EXPERIMENT-1 BINARY CATENATE CHILDREN CONDSTORE CREATE-SPECIAL-USE ESEARCH ESORT LIST-EXTENDED LIST-MYRIGHTS LIST-STATUS MAILBOX-REFERRALS METADATA MOVE MULTIAPPEND NAMESPACE OBJECTID PREVIEW QRESYNC QUOTA RIGHTS=kxten SAVEDATE SEARCH=FUZZY SORT SORT=DISPLAY SPECIAL-USE STATUS=SIZE THREAD=ORDEREDSUBJECT THREAD=REFERENCES UIDPLUS UNSELECT URLAUTH URLAUTH=BINARY WITHIN DIGEST=SHA1 LIST-METADATA NO_ATOMIC_RENAME SCAN SORT=MODSEQ SORT=UID THREAD=REFS X-CREATEDMODSEQ X-REPLICATION X-SIEVE-MAILBOX XLIST XMOVE LOGINDISABLED XCONVERSATIONS COMPRESS=DEFLATE X-QUOTA=STORAGE X-QUOTA=MESSAGE X-QUOTA=X-ANNOTATION-STORAGE X-QUOTA=X-NUM-FOLDERS IDLE APPENDLIMIT=4294967295] Success (tls protection) SESSIONID=
Command 1 took 0.0496 seconds.
CACHE: Using the OCA\Mail\Cache\Cache storage driver.
C: 2 ENABLE QRESYNC
C: 3 NAMESPACE
S: * ENABLED CONDSTORE QRESYNC
S: 2 OK Completed
Command 2 took 0.0418 seconds.
S: * NAMESPACE (("INBOX/" "/")) (("user/" "/")) (("" "/"))
S: 3 OK Completed
Command 3 took 0.085 seconds.
C: 4 LIST () "" (*) RETURN (SUBSCRIBED SPECIAL-USE)
S: * LIST (\Subscribed \HasChildren) "/" INBOX
S: * LIST (\Subscribed \HasNoChildren) "/" INBOX/Archive
.....
S: 4 OK Completed (0.570 secs 14312 calls)
Command 4 took 0.7383 seconds.
Lot of LIST command, seems every subscribed mailboxes, including shared folders completed, see above.
After that a lot of STATUS command happened, seems all:
C: 5 STATUS INBOX (MESSAGES RECENT UIDNEXT UIDVALIDITY UNSEEN)
S: * STATUS INBOX (MESSAGES 8729 RECENT 0 UIDNEXT 23108 UIDVALIDITY 1606248051 UNSEEN 40)
S: 5 OK Completed
Command 5 took 0.0448 seconds.
C: 6 STATUS INBOX/Archive (MESSAGES RECENT UIDNEXT UIDVALIDITY UNSEEN)
S: * STATUS INBOX/Archive (MESSAGES 0 RECENT 0 UIDNEXT 1 UIDVALIDITY 1615624245 UNSEEN 0)
S: 6 OK Completed
Command 6 took 0.0421 seconds.
....
Up to the end or page reload, depending my patience.......
And so on.
* I tried to reach the same imap server from an other Nextcloud, which hosted by Hetzner sharebox, with the same result, while with a dovecot imap server, with much less emails/folders working very well.
Breaking news:
* After several attempts, it seems I got it working, at least now several folders are visible, I can see emails.
* Differences:
* user account changed user1 -> user2 -> and back to user1
* during beeing user2, I turned off to collect information, auto determine importance and show only subscribed folders
* patience, longer than normal
* several php-fpm/apache2 restarts during the attempts (anger management)
## Summary
* It seems, when I have a remote imap server with 14000+ folders and several hundred thousands of emails, mail app seems unresponsive, non-working state.
* During this agony, there is no sign of any errors, as technically it is working in the background.
* When it finishes, the screen of the app remains empty.
Is it an expected behaviour?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the account sync with /usr/bin/php8.1 --define apc.enable_cli=1 -f ./occ mail:account:sync --force -vvv 9, while comparing horde_imap.log and the app behavior for subscribed and full folder lists. Done should mean an account with thousands of folders and hundreds of thousands of emails becomes usable without repeated 10+ minute syncs or an empty screen.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, php
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100