3.18.3 - Federation stopped working - could not find TXT entry for peerDomain

Description:

Federation stopped working after updating from 3.14 to 3.18

Steps to reproduce:

Well ..

Expected behavior:

Federation should work, as it dit before the update

Actual behavior:

No more incoming messages.

Server Setup Information:

• Version of Rocket.Chat Server: 3.18.3
• Operating System: Linux
• Deployment Method: docker
• Number of Running Instances:
• DB Replicaset Oplog:
• NodeJS Version:
• MongoDB Version:

Client Setup Information

• Desktop App or Browser Version:
• Operating System:

Additional context

I can see the following error in the log

rocketchat_1          | Federation ➔ dns.debug search: could not find TXT entry for peerDomain=medelexis.ch - SRV entry found
rocketchat_1          | Federation ➔ dns.debug searchHub: peerDomain=medelexis.ch
rocketchat_1          | Federation ➔ http.debug [GET] https://hub.rocket.chat/api/v1/peers?search=medelexis.ch
rocketchat_1          | server.js:204 Federation ➔ dns.error Error: failed [404] Not Found
rocketchat_1          |     at Object.exports.makeErrorByStatus (packages/http.js:177:10)
rocketchat_1          |     at Request._callback (packages/http.js:141:24)
rocketchat_1          |     at Request.self.callback (/app/bundle/programs/server/npm/node_modules/meteor/http/node_modules/request/request.js:185:22)
rocketchat_1          |     at Request.emit (events.js:314:20)
rocketchat_1          |     at Request.EventEmitter.emit (domain.js:483:12)
rocketchat_1          |     at Request.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/http/node_modules/request/request.js:1161:10)
rocketchat_1          |     at Request.emit (events.js:314:20)
rocketchat_1          |     at Request.EventEmitter.emit (domain.js:483:12)
rocketchat_1          |     at IncomingMessage.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/http/node_modules/request/request.js:1083:12)
rocketchat_1          |     at Object.onceWrapper (events.js:420:28)
rocketchat_1          |     at IncomingMessage.emit (events.js:326:22)
rocketchat_1          |     at IncomingMessage.EventEmitter.emit (domain.js:483:12)
rocketchat_1          |     at endReadableNT (_stream_readable.js:1241:12)
rocketchat_1          |     at processTicksAndRejections (internal/process/task_queues.js:84:21) {
rocketchat_1          |   response: {
rocketchat_1          |     statusCode: 404,
rocketchat_1          |     content: 'Not Found',
rocketchat_1          |     headers: {
rocketchat_1          |       'content-length': '9',
rocketchat_1          |       'content-type': 'text/plain; charset=utf-8',
rocketchat_1          |       date: 'Fri, 03 Dec 2021 09:32:09 GMT',
rocketchat_1          |       etag: 'W/"9-0gXL1ngzMqISxa6S1zx3F4wtLyg"',
rocketchat_1          |       'x-powered-by': 'Express',
rocketchat_1          |       connection: 'close'
rocketchat_1          |     },
rocketchat_1          |     data: null
rocketchat_1          |   }
rocketchat_1          | }
rocketchat_1          | Exception in defer callback: errorClass [Error]: Could not find the peer using DNS or Hub [federation-error-peer-no-found-using-dns]
rocketchat_1          |     at Module.peerNotFoundUsingDNS (app/federation/server/functions/errors.js:10:2)
rocketchat_1          |     at searchHub (app/federation/server/lib/dns.js:59:26)
rocketchat_1          |     at search (app/federation/server/lib/dns.js:138:10)
rocketchat_1          |     at federationRequestToPeer (app/federation/server/lib/http.js:28:38)
rocketchat_1          |     at dispatchEvents (app/federation/server/handler/index.js:65:3)
rocketchat_1          |     at dispatchEvent (app/federation/server/handler/index.js:70:2)
rocketchat_1          |     at app/federation/server/hooks/afterSaveMessage.js:26:2
rocketchat_1          |     at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
rocketchat_1          |  => awaited here:
rocketchat_1          |     at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
rocketchat_1          |     at callback (app/federation/server/hooks/afterSaveMessage.js:33:39)
rocketchat_1          |     at app/callbacks/lib/callbacks.js:160:65
rocketchat_1          |     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)
rocketchat_1          |     at packages/meteor.js:550:25
rocketchat_1          |     at runWithEnvironment (packages/meteor.js:1286:24) {
rocketchat_1          |   isClientSafe: true,
rocketchat_1          |   error: 'federation-error-peer-no-found-using-dns',
rocketchat_1          |   reason: 'Could not find the peer using DNS or Hub',
rocketchat_1          |   details: { method: 'dns.searchHub' },
rocketchat_1          |   error

but dig -t txt rocketchat-public-key.medelexis.ch works as expected, just try for yourself.

The problem occurs in https://github.com/RocketChat/Rocket.Chat/blob/9e7fcb12369b903b542732e4a60fce45c0505163/app/federation/server/lib/dns.js#L137 but could mean anything, as the Exception before this line is simply ommited and not shown!

Relevant logs:

see above

Start at app/federation/server/lib/dns.js around line 137 and trace the DNS and Hub lookup through app/federation/server/lib/http.js, using the reported logs and stack trace as the failure path. Check how the earlier exception is handled and verify the fix by confirming that federation can locate the peer and incoming messages are delivered.