Hotchillee - Slack Import overwrite issue

Context
Hotchillee is looking to move its Digital cycling club communications from Slack to a Rocketchat based app. In doing so it hopes to maintain the user experience as much as possible so as not to confuse its user base while using Rocketchats white labelling capability to give it Hotchillee brand and feel. As part of this Hotchillee looks to retain all the historic chat from Slack to make it as seamless as possible for users transferring to the new platform (similar to when Facebook does a major UX overhaul - all the data remains but the look and feel is different and new features are added but it’s still familiar enough not to alienate users).

Description:

Before December 15th, when importing data from Slack into Rocketchat the import would check and only import new content and not overwrite that content that had already been migrated. However now we can only update once and there is an error if we complete subsequent uploads of data. This compounds the issue above with the content disappearing as we can only check the content works once. It would be great if the old method could come back where incremental imports from Slack could take place allowing for testing and gradual transition.

This was diagnosed as a Slack import issue through the following steps:

Imported the slack export to 17th Dec (the last import before the problems began). The import worked fine, and there were no issues with channels or threads with disappearing content.
Next we tried a full incremental import of slack data to Feb 1st. It complained of a duplicate record and failed to complete the import. That means incremental overlapping imports no longer work.

Steps to reproduce:

1. As described above. Can provide incremental Slack imports if requested to Tim.Hanley@rainmaker.solutions.

Expected behavior:

Incremental Slack imports don't fail.

Actual behavior:

Mongo DB complained of a duplicate record and failed to replicate

Server Setup Information:

• Version of Rocket.Chat Server: 3.11.0
• Operating System:
• Deployment Method:
• Number of Running Instances: 1
• DB Replicaset Oplog:
• NodeJS Version: 12.18.4
• MongoDB Version: 4.2.12
• Apps Engine version: 1.22.1

Client Setup Information

• Desktop App or Browser Version: Both
• Operating System: Mac OS
• Browser - Chrome/Safari

Additional context

Not applicable

Relevant logs:

From server log
I20210205-18:26:46.819(0) server.js:204 Slack Importer ➔ error MongoError: E11000 duplicate key error collection: rocketchat-cmty.rocketchat_message index: id dup key: { _id: "slack-C010RRW7DBK-1590048159-202600-file1" } at Function.create (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/error.js:57:12) at toError (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/utils.js:123:22) at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/operations/common_functions.js:379:39 at handler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/topologies/replset.js:1204:22) at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/pool.js:405:18 at processTicksAndRejections (internal/process/task_queues.js:79:11) { driver: true, index: 0, code: 11000, keyPattern: { _id: 1 }, keyValue: { _id: 'slack-C010RRW7DBK-1590048159-202600-file1' } }

Start with the Slack Importer and the reported MongoDB duplicate-key error for _id slack-C010RRW7DBK-1590048159-202600-file1. Reproduce with overlapping incremental Slack exports if available, comparing the import through the Dec 17 data with the Feb 1 import. Done means a subsequent incremental import completes without duplicate-record failure and existing content remains available.