RocketChat / RocketChat/Rocket.Chat

Duplicate key error on LDAP Sync

Open
#5,938 3 comments 2 reactions 1 assignee View on GitHub

@rodrigok is already working on this.

Since Feb 10, 2017.

area: authentication
Dominant language
TypeScript
Stars
46.1k
Forks
13.9k
Avg merge
3d 3h
Merged PRs (30d)
130

Description

We have encountered this error on LDAP sync with one user. This user was modified in LDAP several times (even changing its DN because of a typo) while not changing its uidNumber which is set as Unique Identifier Field.
LDAP sync after that ends up with an error (E11000 duplicate key error collection). I have tried to completely delete the user from Rocket.Chat but sync after that ends up with the same error. The problem is only solved when I change the uidNumber - but we cannot change the employees number in production LDAP.

Here is the full error log which shows what is going on during the LDAP sync after I have deleted the user from Rocketchat a run "SYNC USERS" (I have renamed the user to "problem.user" for anonymity):

rocketchat_1  | LDAP ➔ Connection.info Init setup
rocketchat_1  | LDAP ➔ Connection.info Connecting ldaps://serverurl:636
rocketchat_1  | LDAP ➔ Connection.info LDAP connected
rocketchat_1  | LDAP ➔ Bind.info Binding admin user xxxxx
rocketchat_1  | LDAP ➔ Search.info Searching user *
rocketchat_1  | LDAP ➔ Search.info Search result count 151
hubot_1       | [Tue Feb 07 2017 15:34:56 GMT+0000 (UTC)] INFO Message received with ID id
hubot_1       | [Tue Feb 07 2017 15:34:56 GMT+0000 (UTC)] INFO Message receive callback id kzErJQTywjz2NLdmW ts Tue Feb 07 2017 15:34:56 GMT+0000 (UTC)
hubot_1       | [Tue Feb 07 2017 15:34:56 GMT+0000 (UTC)] INFO [Incoming] problem.user: problem.user
rocketchat_1  | LDAPSync ➔ info Syncing user data
rocketchat_1  | LDAPSync ➔ info Syncing user avatar
rocketchat_1  | LDAP ➔ Search.info Searching by id 3130343830
rocketchat_1  | LDAP ➔ Search.info Search result count 1
rocketchat_1  | LDAPSync ➔ info Syncing user data
rocketchat_1  | LDAPSync ➔ info Syncing user avatar
rocketchat_1  | LDAP ➔ Search.info Searching by id 3130343834
mongo_1       | 2017-02-07T15:34:56.713+0000 I COMMAND  [conn5] should have chunk: 1 have:0
mongo_1       | c->nextSafe(): { _id: ObjectId('5899bf52a534730001787f6b'), files_id: "problem.user.jpg", n: 0, data: BinData(0, FFD8FFE000104A46494600010101004800480000FFE1151845786966000049492A00080000000D000B0002000D000000AA00000000010900010000002C010000010109000100...) }
mongo_1       | c->nextSafe(): { _id: ObjectId('5899bf52a534730001787f6c'), files_id: "problem.user.jpg", n: 0, data: BinData(0, FFD8FFE000104A46494600010101004800480000FFE1151845786966000049492A00080000000D000B0002000D000000AA00000000010900010000002C010000010109000100...) }
mongo_1       | c->nextSafe(): { _id: ObjectId('5899c41da53473000178810e'), files_id: "problem.user.jpg", n: 0, data: BinData(0, FFD8FFE000104A46494600010101004800480000FFE1151845786966000049492A00080000000D000B0002000D000000AA00000000010900010000002C010000010109000100...) }
mongo_1       | c->nextSafe(): { _id: ObjectId('5899c41da53473000178810f'), files_id: "problem.user.jpg", n: 0, data: BinData(0, FFD8FFE000104A46494600010101004800480000FFE1151845786966000049492A00080000000D000B0002000D000000AA00000000010900010000002C010000010109000100...) }
mongo_1       | c->nextSafe(): { _id: ObjectId('5899c4ada534730001788110'), files_id: "problem.user.jpg", n: 0, data: BinData(0, FFD8FFE000104A46494600010101004800480000FFE1151845786966000049492A00080000000D000B0002000D000000AA00000000010900010000002C010000010109000100...) }
mongo_1       | c->nextSafe(): { _id: ObjectId('5899c4ada534730001788111'), files_id: "problem.user.jpg", n: 0, data: BinData(0, FFD8FFE000104A46494600010101004800480000FFE1151845786966000049492A00080000000D000B0002000D000000AA00000000010900010000002C010000010109000100...) }
mongo_1       | c->nextSafe(): { _id: ObjectId('5899c4b2a534730001788112'), files_id: "problem.user.jpg", n: 0, data: BinData(0, FFD8FFE000104A46494600010101004800480000FFE1151845786966000049492A00080000000D000B0002000D000000AA00000000010900010000002C010000010109000100...) }
mongo_1       | c->nextSafe(): { _id: ObjectId('5899e920a534730001788961'), files_id: "problem.user.jpg", n: 0, data: BinData(0, FFD8FFE000104A46494600010101004800480000FFE1151845786966000049492A00080000000D000B0002000D000000AA00000000010900010000002C010000010109000100...) }
rocketchat_1  | LDAP ➔ Search.info Search result count 1
rocketchat_1  | LDAPSync ➔ info Syncing user data
rocketchat_1  | LDAP ➔ Search.info Searching by id 3130343832
rocketchat_1  | LDAP ➔ Search.info Search result count 1
rocketchat_1  | LDAPSync ➔ info Syncing user data
rocketchat_1  | LDAP ➔ Search.info Searching by id 3130343830
mongo_1       | 2017-02-07T15:34:56.765+0000 I COMMAND  [conn2] should have chunk: 1 have:0
mongo_1       | c->nextSafe(): { _id: ObjectId('5899bf52a534730001787f6b'), files_id: "problem.user.jpg", n: 0, data: BinData(0, FFD8FFE000104A46494600010101004800480000FFE1151845786966000049492A00080000000D000B0002000D000000AA00000000010900010000002C010000010109000100...) }
mongo_1       | c->nextSafe(): { _id: ObjectId('5899bf52a534730001787f6c'), files_id: "problem.user.jpg", n: 0, data: BinData(0, FFD8FFE000104A46494600010101004800480000FFE1151845786966000049492A00080000000D000B0002000D000000AA00000000010900010000002C010000010109000100...) }
mongo_1       | c->nextSafe(): { _id: ObjectId('5899c41da53473000178810e'), files_id: "problem.user.jpg", n: 0, data: BinData(0, FFD8FFE000104A46494600010101004800480000FFE1151845786966000049492A00080000000D000B0002000D000000AA00000000010900010000002C010000010109000100...) }
mongo_1       | c->nextSafe(): { _id: ObjectId('5899c41da53473000178810f'), files_id: "problem.user.jpg", n: 0, data: BinData(0, FFD8FFE000104A46494600010101004800480000FFE1151845786966000049492A00080000000D000B0002000D000000AA00000000010900010000002C010000010109000100...) }
mongo_1       | c->nextSafe(): { _id: ObjectId('5899c4ada534730001788110'), files_id: "problem.user.jpg", n: 0, data: BinData(0, FFD8FFE000104A46494600010101004800480000FFE1151845786966000049492A00080000000D000B0002000D000000AA00000000010900010000002C010000010109000100...) }
mongo_1       | c->nextSafe(): { _id: ObjectId('5899c4ada534730001788111'), files_id: "problem.user.jpg", n: 0, data: BinData(0, FFD8FFE000104A46494600010101004800480000FFE1151845786966000049492A00080000000D000B0002000D000000AA00000000010900010000002C010000010109000100...) }
mongo_1       | c->nextSafe(): { _id: ObjectId('5899c4b2a534730001788112'), files_id: "problem.user.jpg", n: 0, data: BinData(0, FFD8FFE000104A46494600010101004800480000FFE1151845786966000049492A00080000000D000B0002000D000000AA00000000010900010000002C010000010109000100...) }
mongo_1       | c->nextSafe(): { _id: ObjectId('5899e920a534730001788961'), files_id: "problem.user.jpg", n: 0, data: BinData(0, FFD8FFE000104A46494600010101004800480000FFE1151845786966000049492A00080000000D000B0002000D000000AA00000000010900010000002C010000010109000100...) }
mongo_1       | c->nextSafe(): { _id: ObjectId('5899e920a534730001788962'), files_id: "problem.user.jpg", n: 0, data: BinData(0, FFD8FFE000104A46494600010101004800480000FFE1151845786966000049492A00080000000D000B0002000D000000AA00000000010900010000002C010000010109000100...) }
rocketchat_1  | LDAP ➔ Search.info Search result count 1
rocketchat_1  | LDAPSync ➔ info Syncing user data
rocketchat_1  | Exception in callback of async function: MongoError: chunks out of order
rocketchat_1  |   at Function.MongoError.create (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/error.js:31:11)
rocketchat_1  |   at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:483:72
rocketchat_1  |   at authenticateStragglers (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:429:16)
rocketchat_1  |   at [object Object].messageHandler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:463:5)
rocketchat_1  |   at Socket.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/connection.js:317:22)
rocketchat_1  |   at emitOne (events.js:77:13)
rocketchat_1  |   at Socket.emit (events.js:169:7)
rocketchat_1  |   at readableAddChunk (_stream_readable.js:153:18)
rocketchat_1  |   at Socket.Readable.push (_stream_readable.js:111:10)
rocketchat_1  |   at TCP.onread (net.js:537:20)
rocketchat_1  | [Tue Feb 07 2017 15:34:56 GMT+0000 (UTC)] ERROR MongoError: chunks out of order
rocketchat_1  |   at Function.MongoError.create (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/error.js:31:11)
rocketchat_1  |   at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:483:72
rocketchat_1  |   at authenticateStragglers (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:429:16)
rocketchat_1  |   at [object Object].messageHandler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:463:5)
rocketchat_1  |   at Socket.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/connection.js:317:22)
rocketchat_1  |   at emitOne (events.js:77:13)
rocketchat_1  |   at Socket.emit (events.js:169:7)
rocketchat_1  |   at readableAddChunk (_stream_readable.js:153:18)
rocketchat_1  |   at Socket.Readable.push (_stream_readable.js:111:10)
rocketchat_1  |   at TCP.onread (net.js:537:20)
rocketchat_1  | rocketchat_ldap rocketchat_ldap.js:776 LDAPSync ➔ error { [MongoError: E11000 duplicate key error collection: rocketchat.users index: emails.address_1 dup key: { : "problem.user@domain.com" }]
rocketchat_1  |   name: 'MongoError',
rocketchat_1  |   message: 'E11000 duplicate key error collection: rocketchat.users index: emails.address_1 dup key: { : "problem.user@domain.com" }',
rocketchat_1  |   driver: true,
rocketchat_1  |   index: 0,
rocketchat_1  |   code: 11000,
rocketchat_1  |   errmsg: 'E11000 duplicate key error collection: rocketchat.users index: emails.address_1 dup key: { : "problem.user@domain.com" }' }
rocketchat_1  | Exception in callback of async function: MongoError: chunks out of order
rocketchat_1  |   at Function.MongoError.create (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/error.js:31:11)
rocketchat_1  |   at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:483:72
rocketchat_1  |   at authenticateStragglers (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:429:16)
rocketchat_1  |   at [object Object].messageHandler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:463:5)
rocketchat_1  |   at Socket.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/connection.js:347:20)
rocketchat_1  |   at emitOne (events.js:77:13)
rocketchat_1  |   at Socket.emit (events.js:169:7)
rocketchat_1  |   at readableAddChunk (_stream_readable.js:153:18)
rocketchat_1  |   at Socket.Readable.push (_stream_readable.js:111:10)
rocketchat_1  |   at TCP.onread (net.js:537:20)
rocketchat_1  | [Tue Feb 07 2017 15:34:56 GMT+0000 (UTC)] ERROR MongoError: chunks out of order
rocketchat_1  |   at Function.MongoError.create (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/error.js:31:11)
rocketchat_1  |   at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:483:72
rocketchat_1  |   at authenticateStragglers (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:429:16)
rocketchat_1  |   at [object Object].messageHandler (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:463:5)
rocketchat_1  |   at Socket.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/connection.js:347:20)
rocketchat_1  |   at emitOne (events.js:77:13)
rocketchat_1  |   at Socket.emit (events.js:169:7)
rocketchat_1  |   at readableAddChunk (_stream_readable.js:153:18)
rocketchat_1  |   at Socket.Readable.push (_stream_readable.js:111:10)
rocketchat_1  |   at TCP.onread (net.js:537:20)
rocketchat_1  | Exception while invoking method 'ldap_sync_users' MongoError: E11000 duplicate key error collection: rocketchat.users index: emails.address_1 dup key: { : "problem.user@domain.com" }
rocketchat_1  |   at Object.Future.wait (/app/bundle/programs/server/node_modules/fibers/future.js:449:15)
rocketchat_1  |   at MongoConnection.<anonymous> (packages/meteor.js:213:24)
rocketchat_1  |   at MongoConnection.(anonymous function) [as update] (/app/bundle/programs/server/packages/mongo.js:766:49)
rocketchat_1  |   at Object.<anonymous> (packages/matb33_collection-hooks.js:504:27)
rocketchat_1  |   at Object.collection.(anonymous function) [as update] (packages/matb33_collection-hooks.js:146:21)
rocketchat_1  |   at [object Object].update (/app/bundle/programs/server/packages/mongo.js:4007:31)
rocketchat_1  |   at ModelsBaseDb.update (/app/bundle/programs/server/packages/rocketchat_lib.js:8872:32)
rocketchat_1  |   at [object Object].model.update (/app/bundle/programs/server/packages/rocketchat_lib.js:8644:24)
rocketchat_1  |   at ModelsBaseDb.update (/app/bundle/programs/server/packages/rocketchat_lib.js:8872:32)
rocketchat_1  |   at [object Object].model.update (/app/bundle/programs/server/packages/rocketchat_lib.js:8644:24)
rocketchat_1  |   at syncUserData (/app/bundle/programs/server/packages/rocketchat_ldap.js:649:17)
rocketchat_1  |   at /app/bundle/programs/server/packages/rocketchat_ldap.js:770:6
rocketchat_1  |   at Array.forEach (native)
rocketchat_1  |   at Object.forEach (/app/bundle/programs/server/packages/rocketchat_lib.js:8147:50)
rocketchat_1  |   at sync (/app/bundle/programs/server/packages/rocketchat_ldap.js:760:10)
rocketchat_1  |   at [object Object].ldap_sync_users (/app/bundle/programs/server/packages/rocketchat_ldap.js:1089:17)
rocketchat_1  |   at [object Object].methodsMap.(anonymous function) (/app/bundle/programs/server/packages/rocketchat_lib.js:1064:26)
rocketchat_1  |   at maybeAuditArgumentChecks (/app/bundle/programs/server/packages/ddp-server.js:2108:12)
rocketchat_1  |   at /app/bundle/programs/server/packages/ddp-server.js:1042:22
rocketchat_1  |   at [object Object]._.extend.withValue (packages/meteor.js:1122:17)
rocketchat_1  |   at /app/bundle/programs/server/packages/ddp-server.js:1041:43
rocketchat_1  |   at [object Object]._.extend.withValue (packages/meteor.js:1122:17)
rocketchat_1  |   at /app/bundle/programs/server/packages/ddp-server.js:1040:48
rocketchat_1  |   at Session.method (/app/bundle/programs/server/packages/ddp-server.js:1018:23)
rocketchat_1  |   at /app/bundle/programs/server/packages/ddp-server.js:880:89
rocketchat_1  |     - - - - -
rocketchat_1  |   at Function.MongoError.create (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/error.js:31:11)
rocketchat_1  |   at toError (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/utils.js:114:22)
rocketchat_1  |   at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/collection.js:1048:67
rocketchat_1  |   at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:455:18
rocketchat_1  |   at nextTickCallbackWith0Args (node.js:436:9)
rocketchat_1  |   at process._tickDomainCallback (node.js:406:13)

I have dumped the whole mongo and searched for string "problem.user" after deleting it and I have found it in:

Binary file avatars.chunks.bson matches (problem.user.jpg occuring)
Binary file rocketchat__trash.bson matches (problem.user occuring multiple times)

Rocket.Chat Version: 0.51.0
Running Instances: 1
DB Replicaset OpLog: Disabled
Node Version: 4.7.1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.