RocketChat / RocketChat/Rocket.Chat

Rocket 6.4.2 wont start due to meteor errors

Open
#30,838 3 comments 12 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
46.1k
Forks
13.9k
Avg merge
3d 3h
Merged PRs (30d)
130

Description

Description:

Hello, i just had very strange bug with rocketchat deployed in k8s in a single pod
so ive noticed increase in active handlers
image
and pod started to fail probes

95s         Warning   Unhealthy      pod/rocketchat-rocketchat-6596d8bc6d-glvrb   Readiness probe failed: Get "http://10.233.78.209:3000/api/info": read tcp 10.160.192.114:50060->10.233.78.209:3000: read: connection reset by peer

ive checked logs and noticed

Errors like this can cause oplog processing errors.
Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
Future node.js versions will automatically exit the process
=================================
=== UnHandledPromiseRejection ===
errorClass [Error]: Error, too many requests. Please slow down. You must wait 27 seconds before trying this endpoint again. [error-too-many-requests]
    at app/api/server/api.ts:332:10
    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40 {
  isClientSafe: true,
  error: 'error-too-many-requests',
  reason: 'Error, too many requests. Please slow down. You must wait 27 seconds before trying this endpoint again.',
  details: { timeToReset: 26091, seconds: 27 },
  errorType: 'Meteor.Error'
}

ive noticed that mentions method has drastically increased but there was no @all calls in any channel atm
image

so ive tried to restart pod but no help with same error
then i restarted mongodb instance one by one, after 3-5 minutes requests were back to normal and pod started
i dont see any strange behavior of mongodb either
image
except primary was switched but that was after restart of mongodb and its seems to be expected
after its stable again i still see error like

^[[BTypeError: Cannot read property 'authRequired' of undefined
    at packages/rocketchat:restivus/lib/route.js:199:16
    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 => awaited here:
    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
    at packages/rocketchat:restivus/lib/route.js:155:13
    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 => awaited here:
    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
    at packages/rocketchat:restivus/lib/route.js:57:18
    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
{"level":40,"time":"2023-11-02T15:27:59.563Z","pid":1,"hostname":"rocketchat-rocketchat-c4999558c-tjppk","name":"DeprecationWarning","section":"API","msg":"The parameter \"query\" in the endpoint \"/api/v1/users.list\" is deprecated and will be removed on version 7.0.0"}
Error: ENOENT: no such file or directory, stat '/tmp/zipFiles/c0092699-6e89-4bf4-bcc8-71c77a9c4539.zip'
 => awaited here:
    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
    at server/lib/dataExport/uploadZipFile.ts:11:16
    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 => awaited here:
    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
    at server/lib/dataExport/processDataDownloads.ts:232:25
    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40 {
  errno: -2,
  code: 'ENOENT',
  syscall: 'stat',
  path: '/tmp/zipFiles/c0092699-6e89-4bf4-bcc8-71c77a9c4539.zip'

and occasionally get exceptions from server

Error: Can't wait without a fiber
    at Function.wait (/app/bundle/programs/server/node_modules/fibers/future.js:159:9)
    at Future.wait (/app/bundle/programs/server/node_modules/fibers/future.js:448:10)
    at MongoConnection.findOne (packages/mongo/mongo_driver.js:829:84)
    at Collection.findOne (packages/mongo/collection.js:459:29)
    at getUserById (packages/accounts-password/password_server.js:5:51)
    at AccountsServer.Accounts.generateResetToken (packages/accounts-password/password_server.js:402:16)
    at AccountsServer.Accounts.sendEnrollmentEmail (packages/accounts-password/password_server.js:567:14)
    at Immediate.<anonymous> (app/lib/server/functions/setUsername.ts:102:14)
    at processImmediate (internal/timers.js:464:21)
    at process.topLevelDomainCallback (domain.js:152:15)
    at process.callbackTrampoline (internal/async_hooks.js:128:24)

ive no idea why this happens, tho i get a strange feeling thats its just another CE awesome new feature, hope someone will help to figure it out, thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported Kubernetes single-pod failure and the logs around app/api/server/api.ts:332. Then inspect the mentioned Restivus route, data export files, and setUsername.ts paths to determine whether the rate-limit rejection, missing ZIP, and fiber errors share a cause. Done would require a reproducible cause and verified recovery without the reported probe failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, mongodb, node.js, typescript
Domain
backend, databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.