makeplane / makeplane/plane

email: stack_email_notification has no distributed lock - duplicate emails sent on concurrent Celery Beat runs

Open
#9,602 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
59.6k
Forks
5.8k
Avg merge
1d 22h
Merged PRs (30d)
49

Description

Bug Description

stack_email_notification reads unprocessed EmailNotificationLog records, dispatches email sub-tasks, then marks them as processed. There is no distributed lock around this task. On multi-pod deployments where two Celery Beat instances run, or when the task takes longer than its 5-minute schedule, two workers process the same unprocessed records and each dispatches a full set of email sub-tasks - resulting in duplicate emails to all subscribers.

Affected file

pps/api/plane/bgtasks/email_notification_task.py:

`python
email_notifications = EmailNotificationLog.objects.filter(processed_at__isnull=True)... # line 49 - read

...

send_email_notification.delay(...) # lines 76-83 - dispatch sub-tasks (before marking processed)

...

EmailNotificationLog.objects.filter(pk__in=processed_notifications).update(
processed_at=timezone.now()
) # line 84 - marked processed AFTER dispatch
`

The send_email_notification sub-task uses a Redis lock (lines 34-43), but releases it after sending. If the second batch of sub-tasks runs after the 300-second lock TTL expires, they acquire the lock and send again.

Failure scenario

  1. Celery Beat schedules stack_email_notification at T=0 and T=5min.
  2. At T=0, worker 1 reads 50 unprocessed records and starts dispatching sub-tasks.
  3. At T=5min, worker 2 reads the same 50 records (still processed_at IS NULL) and dispatches another 50 sub-tasks.
  4. Both batches complete. Every subscriber receives two copies of each email notification.

Also reproducible with a single Beat instance if the task takes longer than 5 minutes (large notification backlog).

Fix

Mark records as processed (within a SELECT FOR UPDATE lock) before dispatching sub-tasks:

`python
with transaction.atomic():
notifications = EmailNotificationLog.objects.select_for_update(skip_locked=True).filter(
processed_at__isnull=True, ...
)
ids = list(notifications.values_list("pk", flat=True))
EmailNotificationLog.objects.filter(pk__in=ids).update(processed_at=timezone.now())

Now dispatch sub-tasks - records are already marked so a second worker skips them

for notification in EmailNotificationLog.objects.filter(pk__in=ids):
send_email_notification.delay(...)
`

skip_locked=True ensures a second concurrent worker skips already-locked rows rather than waiting.

Environment

Plane develop branch (2026-08-13).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in apps/api/plane/bgtasks/email_notification_task.py, especially the query and dispatch flow around lines 49 and 76-84, and review the existing Redis lock in the send_email_notification sub-task. Verify concurrent scheduled runs cannot select the same unprocessed records and that each notification is dispatched only once; no specific test file is named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
django, python, redis
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.