Push notifications possibly sent twice

Description:

Push notification gateway usage is unexpectedly high, and we suspect double sending of pushes.

Steps to reproduce:

Unknown

Expected behavior:

Push notifications are only sent once.

Actual behavior:

Push notification debug logs show that pushes are sent twice (push IDs anonymized):

I20201029-18:00:25.869(0) Push ➔ debug send message "#REDACTED" to userId REDACTED 
I20201029-18:00:25.870(0) Push ➔ debug send to token {   gcm: 'REDACTED1xkfsasd2231' } 
I20201029-18:00:25.871(0) Push ➔ debug send to token {   gcm: 'REDACTED1xkfsasd2231' } 
I20201029-18:00:25.892(0) Push ➔ debug send to token {   apn: 'REDACTED2so2mcxaa2' } 
I20201029-18:00:25.893(0) Push ➔ debug send to token {   apn: 'REDACTED2so2mcxaa2' } 

On end devices however, push notifications appear only once.

Server Setup Information:

• Version of Rocket.Chat Server: 3.7.0
• Operating System: Ubuntu 20.04
• Deployment Method: docker
• Number of Running Instances: 1
• DB Replicaset Oplog: yes
• MongoDB Version: 4.0

Client Setup Information

Multiple

Additional context

In light of the upcoming paid push notification change, this is critical for us. Some numbers for background:

• about 200 registered users
• 50 active
• 16000 Push notifications this month
• no bots that send lots of messages
• just two broadcast-channels with sporadic messages

generally, I think these numbers are unreasonable and together with the duplicate logs point to double Push sending.
With our medium usage we shouldn't exceed the maximum purchasable push limit, while our user count is way below the recommended 1000 user maximum shown on the pricing page for the free edition.

Relevant logs:

see above.

Start by tracing the push notification gateway path that produces the reported debug logs, using the anonymized GCM and APN token entries as the observed behavior. Determine why each token is logged twice and verify that a notification is sent once per token, adding regression coverage if the relevant test location is identified.