E11000 duplicate key error collection: rocketchat.rocketchat_cron_history index

Description:

I get this on latest (container) after an archive restore of the database with a 3 node replicaset (primary + 2x standby)

MongoServerError: Index build failed: 9770599f-1254-4e54-ae91-484b7212e49f: Collection rocketchat.rocketchat_cron_history ( 8b03ec69-6055-4856-a0f3
-aebd7e165853 ) :: caused by :: E11000 duplicate key error collection: rocketchat.rocketchat_cron_history index: intendedAt_1_name_1 dup key: { int
endedAt: new Date(1682078640000), name: "Generate download files for user data" }

|  Rocket.Chat Version: 6.1.2                                      |
|       NodeJS Version: 14.21.2 - x64                              |
|      MongoDB Version: 5.0.16                                     |
|       MongoDB Engine: wiredTiger                                 |
|             Platform: linux                                      |
|         Process Port: 3000                                       |
|     ReplicaSet OpLog: Enabled                                    |
|          Commit Hash: 7507d14ca0                                 |
|        Commit Branch: HEAD

Prior solution works for me, but its still unclear if this is bad in production

use rocketchat
db.rocketchat_cron_history.remove({})

DeprecationWarning: Collection.remove() is deprecated. Use deleteOne, deleteMany, findOneAndDelete, or bulkWrite.
{ acknowledged: true, deletedCount: 79 

Steps to reproduce:

docker exec $MONGODB_CONTAINER_ID sh -c 'mongodump --archive' > ${MONGODB_RESTORE_LOCATION}/${MONGODB_RESTORE_FILE}

docker exec -e MONGODB_RESTORE_LOCATION=${MONGODB_RESTORE_LOCATION} -e MONGODB_RESTORE_FILE=${MONGODB_RESTORE_FILE} \
        $MONGODB_CONTAINER_ID sh -c 'mongorestore  --archive=${MONGODB_RESTORE_LOCATION}/${MONGODB_RESTORE_FILE} --drop'
        
Then redeploy 2x middleware and 3x mongodb containers

Expected behavior:

Middleware starts cleanly after recovery

Actual behavior:

Middleware refuses to start/crashes

Server Setup Information:

• Version of Rocket.Chat Server: 6.1.2
• Operating System: docker on ubuntu
• Deployment Method: docker
• Number of Running Instances: 2
• DB Replicaset Oplog: 3
• NodeJS Version: 14.21.2
• MongoDB Version:5.0.16

Client Setup Information

n/a

Additional context

n/a

Relevant logs:

[see above] also (https://github.com/RocketChat/Rocket.Chat/issues/12386)

No source file or test is named. Start by reproducing the documented mongodump/mongorestore sequence with the three-node replica set and inspect creation of the rocketchat_cron_history intendedAt_1_name_1 index. Done means middleware recovers and starts cleanly, with the duplicate records handled safely.