[FEATURE REQUEST] Natively support envelopes, e.g., CloudEvents
- Dominant language
- JavaScript
- Stars
- 5.3k
- Forks
- 382
- Avg merge
- 7m
- Merged PRs (30d)
- 4
Description
It's a common practice to use message envelopes in event-driven architectures. Most of the time, these envelopes contain information about the company, the domain context, or other application-specific data. Envelopes are also a great way to standardize messages structure across the whole organization, making it easier to build code around it.
The most common case I've found so far —aside from custom formats— is the usage of CloudEvents. Especially, when used together with Kafka and Avro. Trying to define such a system, comes with confusing decisions:
1. What's should `schemaFormat` look like? Should it be JSON Schema, Avro, or CloudEvents? Everything could be defined using JSON Schema or Avro, but since the "data" of the envelope is Avro, things get confusing here. Is CloudEvents a schema format at all? I don't think so.
1. Should we define the message on the AsyncAPI file as if the envelope wasn't there? Or should we include it somehow? If we include it on every message payload, we'll then have to maintain a lot of repeated information across messages, making it harder to update the envelope on every single message.
1. Should we put this information as a protocol binding? If so, is CloudEvents really a protocol? I don't think so.
1. Should we just use an extension? Probably yes as a starting point but this case is very frequent and would love to have AsyncAPI recognize it as a first-class citizen.
For these reasons, I think it would be interesting to come up with a first-class solution for envelopes in AsyncAPI. This would make people feel everything is integrated better instead of hacking the spec here and there.
#### Example
Please, take this example as an illustration of what I'm trying to accomplish here. By no means, it's a final solution.
```yaml
asyncapi: 2.0.0
channels:
test:
message:
schemaFormat: 'application/vnd.apache.avro;version=1.9.0'
envelope:
format: application/cloudevents # This would tell parsers that it's a CloudEvents envelope. Can be optional.
schema:
type: object
properties:
specversion:
type: string
enum: ['1.0', '1.1'] # In case we want to restrict to specific versions of the CloudEvents spec
myCustomField: # Define custom fields here. Those that are not defined on the CE spec.
type: string
payload:
type: record
# ... more Avro stuff
```
---
### Open Questions/Thoughts
1. What if we could define that CloudEvents (CE) message fields must be placed in the headers or in the payload? CE has its own bindings and, in some of them, they allow you to map the fields to headers or to the payload/body. It would be great to have a way to define that.
2. Use `envelope` + `bindings`. For instance, in the case of CE, we could use a binding called `cloudevents` that will allow us to define how it's used (not used, binary, structured, both). I don't think we want to define that the envelope fields are going to be placed on the headers section because then they would not be an envelope anymore, right? Food for thought tho.
3. Should we support in-house envelopes at all? Or should these people be "penalized" for not using a standard format? In any case, we must consider that even CE allows you to define custom fields and their support is a must.
Contributor guide
Research direction
Start by reviewing the AsyncAPI message fields shown in the example—schemaFormat, envelope, payload, and bindings—and compare them with the listed CloudEvents, Kafka, and Avro questions. The work is complete only when the open design questions have an agreed first-class envelope model and a concrete specification shape.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka
- Domain
- backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100