eiffel-community / eiffel-community/eiffel-intelligence
Resulting document after update is larger than 16777216
- Dominant language
- Java
- Stars
- 12
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
### Description
Using default configuration of eiffel-backend-allevents component (from eiffel-easy2use) with AllEventsRules-Eiffel-Agen-Version.json rule set (need to aggregate each event separately).
Started to receive exceptions such as stated bellow on each event, which seems to cause failure in event subscription handling, although they are still persisted in mongo. Value of 16777216 seems to be maximum BSON document size (16 megabytes).
com.mongodb.MongoCommandException: Command failed with error 17419: 'Resulting document after update is larger than 16777216' on server 10.88.0.2:27017. The full response is { "ok" : 0.0, "errmsg" : "Resulting document after update is larger than 16777216", "code" : 17419, "codeName" : "Location17419" }
at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:164) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:295) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:255) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:98) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:441) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:80) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:189) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:264) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:126) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:118) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.operation.CommandOperationHelper$3.call(CommandOperationHelper.java:436) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.operation.OperationHelper.withReleasableConnection(OperationHelper.java:433) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.operation.CommandOperationHelper.executeRetryableCommand(CommandOperationHelper.java:429) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.operation.BaseFindAndModifyOperation.execute(BaseFindAndModifyOperation.java:33) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.operation.FindAndUpdateOperation.execute(FindAndUpdateOperation.java:58) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.Mongo$3.execute(Mongo.java:837) ~[mongodb-driver-3.6.4.jar:na]
at com.mongodb.MongoCollectionImpl.executeFindOneAndUpdate(MongoCollectionImpl.java:765) ~[mongodb-driver-3.6.4.jar:na]
at com.mongodb.MongoCollectionImpl.findOneAndUpdate(MongoCollectionImpl.java:748) ~[mongodb-driver-3.6.4.jar:na]
at com.mongodb.MongoCollectionImpl.findOneAndUpdate(MongoCollectionImpl.java:743) ~[mongodb-driver-3.6.4.jar:na]
at com.ericsson.ei.mongo.MongoDBHandler.doFindAndModify(MongoDBHandler.java:320) ~[classes/:3.1.0]
at com.ericsson.ei.mongo.MongoDBHandler.findAndModify(MongoDBHandler.java:188) ~[classes/:3.1.0]
at com.ericsson.ei.subscription.SubscriptionRepeatDbHandler.updateExistingMatchedSubscriptionWithAggrObjId(SubscriptionRepeatDbHandler.java:153) [classes/:3.1.0]
at com.ericsson.ei.subscription.SubscriptionRepeatDbHandler.addMatchedAggrObjToSubscriptionId(SubscriptionRepeatDbHandler.java:82) [classes/:3.1.0]
at com.ericsson.ei.subscription.RunSubscription.runSubscriptionOnObject(RunSubscription.java:129) [classes/:3.1.0]
at com.ericsson.ei.subscription.SubscriptionHandler.extractConditions(SubscriptionHandler.java:105) [classes/:3.1.0]
at com.ericsson.ei.subscription.SubscriptionHandler.lambda$checkSubscriptionForObject$0(SubscriptionHandler.java:80) [classes/:3.1.0]
at java.util.ArrayList.forEach(ArrayList.java:1257) ~[na:1.8.0_181]
at com.ericsson.ei.subscription.SubscriptionHandler.checkSubscriptionForObject(SubscriptionHandler.java:79) [classes/:3.1.0]
at com.ericsson.ei.handlers.ObjectHandler.postInsertActions(ObjectHandler.java:303) ~[classes/:3.1.0]
at com.ericsson.ei.handlers.ObjectHandler.insertObject(ObjectHandler.java:108) ~[classes/:3.1.0]
at com.ericsson.ei.handlers.ObjectHandler.insertObject(ObjectHandler.java:113) ~[classes/:3.1.0]
at com.ericsson.ei.jsonmerge.MergeHandler.addNewObject(MergeHandler.java:276) ~[classes/:3.1.0]
at com.ericsson.ei.handlers.ExtractionHandler.runExtraction(ExtractionHandler.java:89) ~[classes/:3.1.0]
at com.ericsson.ei.rules.IdRulesHandler.runIdRules(IdRulesHandler.java:63) ~[classes/:3.1.0]
at com.ericsson.ei.handlers.EventHandler.eventReceived(EventHandler.java:57) ~[classes/:3.1.0]
at com.ericsson.ei.handlers.EventHandler.onMessage(EventHandler.java:69) ~[classes/:3.1.0]
at com.ericsson.ei.handlers.EventHandler$$FastClassBySpringCGLIB$$e31fe186.invoke() ~[classes/:3.1.0]
### Motivation
Avoid failed event subscription handling.
### Exemplification
### Benefits
Persistence of un-processed events.
### Possible Drawbacks
Contributor guide
Research direction
Start with SubscriptionRepeatDbHandler.updateExistingMatchedSubscriptionWithAggrObjId and its call to MongoDBHandler.findAndModify, using the AllEventsRules-Eiffel-Agen-Version.json configuration to reproduce the 16 MB MongoDB update failure. Trace how matched aggregation object IDs accumulate for each event. Done means event subscription handling no longer fails when the resulting document reaches MongoDB's size limit, while events remain persisted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mongodb
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100