openedx / openedx/openedx-events
[Abstraction] Decide how to handle key deserialization when key schema might vary
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13
- Forks
- 33
- Avg merge
- 14h 37m
- Merged PRs (30d)
- 4
Description
In the current event bus code, it is possible for signals to be broadcast as events using varying key IDs on the same topic. For example, the CERTIFICATE_CREATED signal could be emitted as events with a key taken from certificate.user.id, or a key taken from certificate.course.course_key. This presents a problem for deserialization: One of those is a number, and the other is a string; deserialization of the key will fail if the key schema is not knowable from the event.
I can see several possible solutions:
- By documented convention, never use the same topic to send events for the same signal but with differing key-fields
- Producer and consumer code would continue to explicitly specify "produce on/listen to topic X for signal Y with key-field Z"
- Kind of fragile and redundant
- Have the producer include the key-field in the event headers (
"key_field": "certificate.user.id") so that the consumer can tell how to deserialize the key.- Self-documenting in a way that should be pretty robust; would handle schema evolution very easily.
- Specify the key-field on the OpenEdxPublicSignal itself, so that signals can always only be sent with the same key-field.
- Less flexible, although there's a migration path for changing that.
- Simpler.
- Conflates signals and events maybe a little more than we want.
- Not clear how we'd handle composite keys such as we'd likely need for certificates -- combine the user and course identifiers by using a list of key-fields? Have a blank key?
- Obviate the whole issue by never deserializing keys, and just emitting them as bytestrings when using them in logging code and similar.
- It's possible we don't really have a use for keys beyond debugging and other observability.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing key serialization and deserialization in the event bus, then inspect how OpenEdxPublicSignal and event headers are used. Compare the listed approaches, including varying numeric, string, and composite keys; done means selecting and documenting a compatible behavior for producers, consumers, and observability.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100