Database inconsistencies for scim / saml users between spar, brig?

In rare cases after production availability issues, you may get 409 conflict responses for creating new users. Searching for those users with curl on brig or on ES will yield no results.

The 409 conflict can have two possible causes:

1. SAML NameID (externalId in scim, usually email address); symptom: the error message contains the phrase "externalId is already taken".
2. Wire handle (userName in scim); symptom: error message contains the user handle in the reason phrase.

The first one is more likely. To confirm, talk to spar's cassandra:

$ /opt/cassandra/bin/cqlsh $(hostname -i) | tee table.dump

> select * from spar.user where issuer='<entity id of the IdP>'

Now press <enter> a few times until you've seen the entire output, then:

$ grep $externaiId table.dump

If this doesn't yield anything, you don't have ruled out case 1.

If it does yield one line, you have a user id. If that user doesn't exist in brig, you have confirmed 1.

Work-around:

delete * from spar.user where issuer='<what you entered above>' and sso_id='<what you found above>'

Fix: coming up!

Start by reproducing the 409 conflict and use the documented spar Cassandra query to inspect the affected issuer and SAML user records, then compare them with brig and Elasticsearch. Trace how the stale spar.user record causes the externalId or userName conflict. Done means inconsistent users no longer produce this conflict and the spar, brig, and Elasticsearch records agree.