RocketChat / RocketChat/Rocket.Chat
Rocket 6.4.2 wont start due to meteor errors
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 46.1k
- Forks
- 13.9k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 130
Description
Description:
Hello, i just had very strange bug with rocketchat deployed in k8s in a single pod
so ive noticed increase in active handlers
and pod started to fail probes
95s Warning Unhealthy pod/rocketchat-rocketchat-6596d8bc6d-glvrb Readiness probe failed: Get "http://10.233.78.209:3000/api/info": read tcp 10.160.192.114:50060->10.233.78.209:3000: read: connection reset by peer
ive checked logs and noticed
Errors like this can cause oplog processing errors.
Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
Future node.js versions will automatically exit the process
=================================
=== UnHandledPromiseRejection ===
errorClass [Error]: Error, too many requests. Please slow down. You must wait 27 seconds before trying this endpoint again. [error-too-many-requests]
at app/api/server/api.ts:332:10
at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40 {
isClientSafe: true,
error: 'error-too-many-requests',
reason: 'Error, too many requests. Please slow down. You must wait 27 seconds before trying this endpoint again.',
details: { timeToReset: 26091, seconds: 27 },
errorType: 'Meteor.Error'
}
ive noticed that mentions method has drastically increased but there was no @all calls in any channel atm
so ive tried to restart pod but no help with same error
then i restarted mongodb instance one by one, after 3-5 minutes requests were back to normal and pod started
i dont see any strange behavior of mongodb either
except primary was switched but that was after restart of mongodb and its seems to be expected
after its stable again i still see error like
^[[BTypeError: Cannot read property 'authRequired' of undefined
at packages/rocketchat:restivus/lib/route.js:199:16
at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
=> awaited here:
at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
at packages/rocketchat:restivus/lib/route.js:155:13
at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
=> awaited here:
at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
at packages/rocketchat:restivus/lib/route.js:57:18
at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
{"level":40,"time":"2023-11-02T15:27:59.563Z","pid":1,"hostname":"rocketchat-rocketchat-c4999558c-tjppk","name":"DeprecationWarning","section":"API","msg":"The parameter \"query\" in the endpoint \"/api/v1/users.list\" is deprecated and will be removed on version 7.0.0"}
Error: ENOENT: no such file or directory, stat '/tmp/zipFiles/c0092699-6e89-4bf4-bcc8-71c77a9c4539.zip'
=> awaited here:
at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
at server/lib/dataExport/uploadZipFile.ts:11:16
at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
=> awaited here:
at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
at server/lib/dataExport/processDataDownloads.ts:232:25
at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40 {
errno: -2,
code: 'ENOENT',
syscall: 'stat',
path: '/tmp/zipFiles/c0092699-6e89-4bf4-bcc8-71c77a9c4539.zip'
and occasionally get exceptions from server
Error: Can't wait without a fiber
at Function.wait (/app/bundle/programs/server/node_modules/fibers/future.js:159:9)
at Future.wait (/app/bundle/programs/server/node_modules/fibers/future.js:448:10)
at MongoConnection.findOne (packages/mongo/mongo_driver.js:829:84)
at Collection.findOne (packages/mongo/collection.js:459:29)
at getUserById (packages/accounts-password/password_server.js:5:51)
at AccountsServer.Accounts.generateResetToken (packages/accounts-password/password_server.js:402:16)
at AccountsServer.Accounts.sendEnrollmentEmail (packages/accounts-password/password_server.js:567:14)
at Immediate.<anonymous> (app/lib/server/functions/setUsername.ts:102:14)
at processImmediate (internal/timers.js:464:21)
at process.topLevelDomainCallback (domain.js:152:15)
at process.callbackTrampoline (internal/async_hooks.js:128:24)
ive no idea why this happens, tho i get a strange feeling thats its just another CE awesome new feature, hope someone will help to figure it out, thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported Kubernetes single-pod failure and the logs around app/api/server/api.ts:332. Then inspect the mentioned Restivus route, data export files, and setUsername.ts paths to determine whether the rate-limit rejection, missing ZIP, and fiber errors share a cause. Done would require a reproducible cause and verified recovery without the reported probe failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, mongodb, node.js, typescript
- Domain
- backend, databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100