eiffel-community / eiffel-community/eiffel-intelligence

Resulting document after update is larger than 16777216

Open
#484 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
12
Forks
78
PR merge metrics
No merged PRs in 30d

Description

### Description
Using default configuration of eiffel-backend-allevents component (from eiffel-easy2use) with AllEventsRules-Eiffel-Agen-Version.json rule set (need to aggregate each event separately).

Started to receive exceptions such as stated bellow on each event, which seems to cause failure in event subscription handling, although they are still persisted in mongo. Value of 16777216 seems to be maximum BSON document size (16 megabytes).

com.mongodb.MongoCommandException: Command failed with error 17419: 'Resulting document after update is larger than 16777216' on server 10.88.0.2:27017. The full response is { "ok" : 0.0, "errmsg" : "Resulting document after update is larger than 16777216", "code" : 17419, "codeName" : "Location17419" }
at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:164) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:295) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:255) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:98) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:441) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:80) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:189) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:264) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:126) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:118) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.operation.CommandOperationHelper$3.call(CommandOperationHelper.java:436) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.operation.OperationHelper.withReleasableConnection(OperationHelper.java:433) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.operation.CommandOperationHelper.executeRetryableCommand(CommandOperationHelper.java:429) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.operation.BaseFindAndModifyOperation.execute(BaseFindAndModifyOperation.java:33) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.operation.FindAndUpdateOperation.execute(FindAndUpdateOperation.java:58) ~[mongodb-driver-core-3.6.4.jar:na]
at com.mongodb.Mongo$3.execute(Mongo.java:837) ~[mongodb-driver-3.6.4.jar:na]
at com.mongodb.MongoCollectionImpl.executeFindOneAndUpdate(MongoCollectionImpl.java:765) ~[mongodb-driver-3.6.4.jar:na]
at com.mongodb.MongoCollectionImpl.findOneAndUpdate(MongoCollectionImpl.java:748) ~[mongodb-driver-3.6.4.jar:na]
at com.mongodb.MongoCollectionImpl.findOneAndUpdate(MongoCollectionImpl.java:743) ~[mongodb-driver-3.6.4.jar:na]
at com.ericsson.ei.mongo.MongoDBHandler.doFindAndModify(MongoDBHandler.java:320) ~[classes/:3.1.0]
at com.ericsson.ei.mongo.MongoDBHandler.findAndModify(MongoDBHandler.java:188) ~[classes/:3.1.0]
at com.ericsson.ei.subscription.SubscriptionRepeatDbHandler.updateExistingMatchedSubscriptionWithAggrObjId(SubscriptionRepeatDbHandler.java:153) [classes/:3.1.0]
at com.ericsson.ei.subscription.SubscriptionRepeatDbHandler.addMatchedAggrObjToSubscriptionId(SubscriptionRepeatDbHandler.java:82) [classes/:3.1.0]
at com.ericsson.ei.subscription.RunSubscription.runSubscriptionOnObject(RunSubscription.java:129) [classes/:3.1.0]
at com.ericsson.ei.subscription.SubscriptionHandler.extractConditions(SubscriptionHandler.java:105) [classes/:3.1.0]
at com.ericsson.ei.subscription.SubscriptionHandler.lambda$checkSubscriptionForObject$0(SubscriptionHandler.java:80) [classes/:3.1.0]
at java.util.ArrayList.forEach(ArrayList.java:1257) ~[na:1.8.0_181]
at com.ericsson.ei.subscription.SubscriptionHandler.checkSubscriptionForObject(SubscriptionHandler.java:79) [classes/:3.1.0]
at com.ericsson.ei.handlers.ObjectHandler.postInsertActions(ObjectHandler.java:303) ~[classes/:3.1.0]
at com.ericsson.ei.handlers.ObjectHandler.insertObject(ObjectHandler.java:108) ~[classes/:3.1.0]
at com.ericsson.ei.handlers.ObjectHandler.insertObject(ObjectHandler.java:113) ~[classes/:3.1.0]
at com.ericsson.ei.jsonmerge.MergeHandler.addNewObject(MergeHandler.java:276) ~[classes/:3.1.0]
at com.ericsson.ei.handlers.ExtractionHandler.runExtraction(ExtractionHandler.java:89) ~[classes/:3.1.0]
at com.ericsson.ei.rules.IdRulesHandler.runIdRules(IdRulesHandler.java:63) ~[classes/:3.1.0]
at com.ericsson.ei.handlers.EventHandler.eventReceived(EventHandler.java:57) ~[classes/:3.1.0]
at com.ericsson.ei.handlers.EventHandler.onMessage(EventHandler.java:69) ~[classes/:3.1.0]
at com.ericsson.ei.handlers.EventHandler$$FastClassBySpringCGLIB$$e31fe186.invoke() ~[classes/:3.1.0]

### Motivation
Avoid failed event subscription handling.

### Exemplification

### Benefits
Persistence of un-processed events.

### Possible Drawbacks

Contributor guide

Open the contributing guide

Research direction

Start with SubscriptionRepeatDbHandler.updateExistingMatchedSubscriptionWithAggrObjId and its call to MongoDBHandler.findAndModify, using the AllEventsRules-Eiffel-Agen-Version.json configuration to reproduce the 16 MB MongoDB update failure. Trace how matched aggregation object IDs accumulate for each event. Done means event subscription handling no longer fails when the resulting document reaches MongoDB's size limit, while events remain persisted.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mongodb
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.