st2auth failed create token mongoengine.errors.NotUniqueError: Tried to save duplicate unique keys

SUMMARY

Provide a quick summary of your bug report.

We get a lot Events in Nagios that we route to st2 using the stackstorm-nagios pack. This works most the time very well. But in scale we get Problems than there a many parallel events.

In this case send some events failed.
We can see that in the st2auth log:

2023-10-24 06:56:07,215 139812116836016 ERROR base [-] Conflict while trying to save in DB.
Traceback (most recent call last):
File "/opt/stackstorm/st2/lib/python3.8/site-packages/mongoengine/document.py", line 398, in save
object_id = self._save_create(doc, force_insert, write_concern)
File "/opt/stackstorm/st2/lib/python3.8/site-packages/mongoengine/document.py", line 463, in _save_create
object_id = wc_collection.insert_one(doc).inserted_id
File "/opt/stackstorm/st2/lib/python3.8/site-packages/pymongo/collection.py", line 698, in insert_one
self._insert(document,
File "/opt/stackstorm/st2/lib/python3.8/site-packages/pymongo/collection.py", line 613, in _insert
return self._insert_one(
File "/opt/stackstorm/st2/lib/python3.8/site-packages/pymongo/collection.py", line 602, in _insert_one
self.__database.client._retryable_write(
File "/opt/stackstorm/st2/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1498, in _retryable_write
return self._retry_with_session(retryable, func, s, None)
File "/opt/stackstorm/st2/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1384, in _retry_with_session
return self._retry_internal(retryable, func, session, bulk)
File "/opt/stackstorm/st2/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1416, in _retry_internal
return func(session, sock_info, retryable)
File "/opt/stackstorm/st2/lib/python3.8/site-packages/pymongo/collection.py", line 600, in _insert_command
_check_write_command_response(result)
File "/opt/stackstorm/st2/lib/python3.8/site-packages/pymongo/helpers.py", line 226, in _check_write_command_response
_raise_last_write_error(write_errors)
File "/opt/stackstorm/st2/lib/python3.8/site-packages/pymongo/helpers.py", line 207, in _raise_last_write_error
raise DuplicateKeyError(error.get("errmsg"), 11000, error)
pymongo.errors.DuplicateKeyError: E11000 duplicate key error collection: st2.user_role_assignment_d_b index: role_1_user_1_source_1 dup key: { : "admin", : "user-nagios", : "mappings/sensorsldap.yaml" }, full error: {'index': 0, 'code': 11000, 'errmsg': 'E11000 duplicate key error collection: st2.user_role_assignment_d_b index: role_1_user_1_source_1 dup key: { : "admin", : "user-nagios", : "mappings/sensorsldap.yaml" }'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/persistence/base.py", line 185, in add_or_update
model_object = cls._get_impl().add_or_update(model_object, validate=True)
File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/models/db/init.py", line 606, in add_or_update
instance.save(validate=validate)
File "/opt/stackstorm/st2/lib/python3.8/site-packages/mongoengine/document.py", line 421, in save
raise NotUniqueError(message % err)
mongoengine.errors.NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.user_role_assignment_d_b index: role_1_user_1_source_1 dup key: { : "admin", : "user-nagios", : "mappings/sensorsldap.yaml" }, full error: {'index': 0, 'code': 11000, 'errmsg': 'E11000 duplicate key error collection: st2.user_role_assignment_d_b index: role_1_user_1_source_1 dup key: { : "admin", : "user-nagios", : "mappings/sensorsldap.yaml" }'})

STACKSTORM VERSION

Paste the output of st2 --version:
st2 3.8.0, on Python 3.8.16

OS, environment, install method

Post what OS you are running this on, along with any other relevant information/

• e.g. Docker, Vagrant, Kubernetes, etc. Describe how you installed ST2
• e.g. one-line install, custom install, etc -->

RHEL 8

Steps to reproduce the problem

Show how to reproduce the problem, using a minimal test-case. Make sure to include any content
(pack content - workflows, actions, etc.) which are needed to reproduce the problem.

Clone the stackstorm-nagios pack
Go to etc dir in that pack.
Edit the config yaml with user pasword.
Create a bashscript to run the Nagios Python Script parallel.
Like that:

python3 ./st2service_handler.py st2service_handler.yaml 44534 3 WARNING HARD "/var/log" 4 host-name --verbose & python3 ./st2service_handler.py st2service_handler.yaml 44534 3 WARNING HARD "/var/log" 4 host-name --verbose & python3 ./st2service_handler.py st2service_handler.yaml 44534 3 WARNING HARD "/var/log" 4 host-name --verbose & python3 ./st2service_handler.py st2service_handler.yaml 44534 3 WARNING HARD "/var/log" 4 host-name --verbose &

Expected Results

What did you expect to happen when running the steps above?

I expect hat all Events come to ST2.

Actual Results

What happened? What output did you get?

We get always some 403 errors.
But at the same time some other Events of Nagios come in.

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

Thank you for your Advice

Reproduce the parallel requests with stackstorm-nagios's st2service_handler.py and inspect the st2auth traceback through st2common/persistence/base.py and st2common/models/db/init.py. Trace the user role assignment and token-creation path, then verify that concurrent Nagios events no longer produce duplicate-key errors or 403 responses and that all events reach ST2.