openedx / openedx/openedx-events

send_event loses observability into receiver exceptions

Open
#136 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13
Forks
33
Avg merge
14h 37m
Merged PRs (30d)
4

Description

send_robust catches exceptions raised by receivers and returns them (which are then passed through send_event as well), but this means that errors don't make it into New Relic (or other telemetry). This is a general problem with Django, but it may be particularly troublesome in the context of signal teleportation on the event bus, as the receiver may begin failing (perhaps due to a new bug or a change in the data) and the errors would not raise alerts or be available for debugging. Being able to detect a broken producer/consumer pair and fix it will be important.

The solution here may be to set a custom attribute, call NR's notice_exception, or even ask NR to instrument send_robust (although that would only solve the issue for NR users).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing send_event and send_robust to understand how receiver exceptions are caught and returned. Review the project's existing telemetry integration, if any, before choosing between a custom attribute, notice_exception, or instrumentation. Done means receiver failures become visible to New Relic or other supported telemetry without changing the signal result handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.