Account sync performance issue

Steps to reproduce

1. Configure several imap accounts with 2GB of emails under MS Exchange 2007
2. Let Nextcloud perform default background sync of these account with system cron
3. See multiple cron processes on the server, as it takes more than 30 minutes to execute

Expected behavior

Background sync of mail accounts should be quicker

Actual behavior

A single account with 2GB of mails sync can take several minutes to be executed. Even if we have just synced it a few minutes ago.
When you have 400 accounts and each one take 3 minutes to sync, the background job may takes hours to finish and the nextcloud cronjob cannot be finished before the next one begins, five minutes later.

The old mail server (Exchange 2007) may be partly responsible of the slow sync, but other clients like Thunderbird, Outlook or some other webmails (SnappyMail) are displaying this very same mailboxes instantly after the first sync. Each future refresh takes then a few milliseconds to perform.

Syncing with occ command does not show relevant informations (see below)
Disabling the Mail app allow the nextcloud cronjob to be executed in a couple of minutes as it should do for this instance scale I guess.

Mail app version

3.2.0

Mailserver or service

Exchange 2007

Operating system

Debian 11

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database

MariaDB

Additional info

root@cloud:/var/www/nextcloud# time sudo -u www-data php8.1 ./occ mail:account:sync -vvv 190
[debug] account is up to date, skipping mailbox sync
[debug] Skipping mailbox sync for 12558
[debug] Skipping mailbox sync for 12560
[debug] Skipping mailbox sync for 12561
[debug] Skipping mailbox sync for 12562
[debug] Skipping mailbox sync for 12563
[debug] Skipping mailbox sync for 12564
[debug] Skipping mailbox sync for 12565
[debug] Skipping mailbox sync for 12566
[debug] Skipping mailbox sync for 16009
[debug] Skipping mailbox sync for 16037
[debug] Skipping mailbox sync for 16369
[debug] Skipping mailbox sync for 16073
[debug] Skipping mailbox sync for 16021
[debug] Skipping mailbox sync for 16022
[debug] Skipping mailbox sync for 16065
[debug] Skipping mailbox sync for 16011
[debug] Skipping mailbox sync for 16019
[debug] Skipping mailbox sync for 16028
[debug] Skipping mailbox sync for 16023
[debug] Skipping mailbox sync for 16018
[debug] Skipping mailbox sync for 16029
[debug] Skipping mailbox sync for 16020
[debug] Skipping mailbox sync for 16061
[debug] Skipping mailbox sync for 16062
[debug] Skipping mailbox sync for 16034
[debug] Skipping mailbox sync for 16015
[debug] Skipping mailbox sync for 16064
[debug] Skipping mailbox sync for 16066
[debug] Skipping mailbox sync for 16068
[debug] Skipping mailbox sync for 16010
[debug] Skipping mailbox sync for 16014
[debug] Skipping mailbox sync for 16025
[debug] Skipping mailbox sync for 16012
[debug] Skipping mailbox sync for 16060
[debug] Skipping mailbox sync for 16026
[debug] Skipping mailbox sync for 16027
[debug] Skipping mailbox sync for 16013
[debug] Skipping mailbox sync for 16069
[debug] Skipping mailbox sync for 16070
[debug] Skipping mailbox sync for 16071
[debug] Skipping mailbox sync for 16072
[debug] Skipping mailbox sync for 16030
[debug] Skipping mailbox sync for 16035
[debug] Skipping mailbox sync for 16036
[debug] Skipping mailbox sync for 16031
[debug] Skipping mailbox sync for 16032
[debug] Syncing 12559
[debug] Locking mailbox 12559 for new messages sync
[debug] Locking mailbox 12559 for changed messages sync
[debug] Locking mailbox 12559 for vanished messages sync
[debug] Running partial sync for 12559
[debug] partial sync 190:INBOX - get all known UIDs took 0s. 57/57MB memory used
[debug] partial sync 190:INBOX - get new messages via Horde took 2s. 58/58MB memory used
[debug] partial sync 190:INBOX - classified a chunk of new messages took 4s. 59/59MB memory used
[debug] partial sync 190:INBOX - persist new messages took 0s. 59/59MB memory used
[debug] partial sync 190:INBOX - get changed messages via Horde took 179s. 81/83MB memory used
[debug] partial sync 190:INBOX - persist changed messages took 2s. 81/84MB memory used
[debug] partial sync 190:INBOX - get vanished messages via Horde took 0s. 78/84MB memory used
[debug] partial sync 190:INBOX - delete vanished messages took 0s. 78/84MB memory used
**[debug] partial sync 190:INBOX took 187s**
[debug] Unlocking mailbox 12559 from vanished messages sync
[debug] Unlocking mailbox 12559 from changed messages sync
[debug] Unlocking mailbox 12559 from new messages sync
[debug] Skipping mailbox sync for 12570
[debug] Skipping mailbox sync for 12571
[debug] Skipping mailbox sync for 12572
[debug] Skipping mailbox sync for 12573
[debug] Skipping mailbox sync for 12574
[debug] Skipping mailbox sync for 12575
[debug] Skipping mailbox sync for 12576
[debug] Skipping mailbox sync for 12567
[debug] Skipping mailbox sync for 12568
[debug] Building threads for account 190
[debug] Account 190 has 2645 messages with threading information
[debug] Threading 2645 messages - build ID table took 0s. 81/84MB memory used
[debug] Threading 2645 messages - build root container took 0s. 81/84MB memory used
[debug] Threading 2645 messages - free ID table took 0s. 81/84MB memory used
[debug] Threading 2645 messages - prune containers took 0s. 81/84MB memory used
[debug] Threading 2645 messages - group by subject took 0s. 81/84MB memory used
[debug] Threading 2645 messages took 0s
[debug] Account 190 has 2308 threads
[debug] Account 190 has 3 messages with a new thread IDs
[debug] Chunk of 500 messages updated
83MB of memory used

real    3m10,472s
user    0m14,859s
sys     0m9,391s

Look at the execution duration column of the oc_jobs table :

Resulting in nextcloud cron job launched at 15:15 is already running at 16:04 :

Start with the occ mail:account:sync -vvv 190 entry point and its timing output, especially the 179-second Horde step during partial sync of mailbox 12559. Review the account-sync path and the oc_jobs execution durations described in the report. Done means the cause of the repeated slow sync is identified and background account synchronization no longer causes overlapping long-running cron jobs.