Azure / Azure/azure-functions-host
Multiple RuleDescriptions per subscription
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
When adding multiple `RuleDescription` to a topic subscription, all added rules, and their filters, use a logical AND behavior. Why is this the case? Wouldn't it make more sense to have logical AND between filter properties of a rule, and then a logical OR between the separate rules (per subscription)? Or at least be able to indicate what the desired behavior should be, AND/OR, between properties and especially rules?
We use a single topic, `topics.employees`, where we have many different subscriptions, e.g. `EmployeeCreated`. The problem with this is, that we have many different types of customers using our primary system, and each of our customers have very specific and unique use-cases/needs. To avoid having to integrate many smaller, unique use-cases for external systems into our primary system that only one or maybe two of our customers need, we want to be able to instead build single Azure Functions per external system, that our customers might be in need of. These external systems have so far primarily been portals for our customer's employees, and thus, we have the topic called `topics.employees` in which we have a multitude of subscriptions per external system - e.g.: `ExternalSystemA_EmployeeCreated`. We'd want to be able to add rules to this subscription in such a fashion that it'd allow for processing only emitted messages from the certain customers who in fact have a need for the message to be processed by `ExternalSystemA_EmployeeCreated`.
Continuing from the example above, we have a main system where we emit an `EmployeeChangeMessage`, which is sent to our `topics.employees` as a brokered message. To the message we append the following properties: `MainID`, `SecondID`, `PayloadType`.
* `MainID` is a unique ID for a chain (think Wallmart as a whole)
* `SecondID` is a specific department of that particular chain (think an individual, unique Wallmart store - we do this, as some of our customers have use-cases where only some of their departments are in need of the external system)
* The `PayloadType` is used to distinguish between the different types of payloads we send from the main system (other than `EmployeeChangeMessage`, we also emit messages such as `SettingsChanged` etc.)
Because we want the main system to be as loosely coupled as possible, we do _not_ want it to know of the specific needs of each department, e.g.: "only emit the `EmployeeChangedMessage`, if one of these external systems are enabled...", or "add this extra property because this particular department has a certain need for a specific external system to process the message".
Thus, we want the function apps to be able to distinguish between the messages sent to the topic by (what we thought would be possible) having multiple rules for each external system's subscription, enabling us to filter messages before they reach the function app. Such as, Customer A has Departments A, B and C. Only emitted `EmployeeCreated` messages coming from Department A and C, should reach the subscription `ExternalSystemA_EmployeeCreated`.
#### Investigative information
Using function app version 2
Because I'm not the boss of my company, and we make our living by offering software solutions to our customers, and the fact that we're a small company who can't afford others capitalizing on our work, I'll refrain from sharing any code insight at this point, unless you find it _extremely_ important - although I find it hard to see why, as this is more question of design than actual locating a bug in our code.
#### Repro steps
1. Delete the initial default rule
2. Add a rule that will match against e.g. `MainID = 100` **AND** `SecondID = 200` **AND** `PayloadType = EmployeeCreated`
3. Add a rule that will match against e.g. `MainID = 101` **AND** `SecondID = 201` **AND** `PayloadType = EmployeeCreated`
(All steps done on the same subscription, of course)
```csharp
var filterProperties = new Dictionary
{
{"MainID", "100" },
{"SecondID", "200" },
{"PayloadType", "EmployeeCreated"}
};
var filter = new CorrelationFilter();
foreach (var pair in filterProperties)
filter.Properties.Add(pair.Key, pair.Value);
var rule = new RuleDescription(ruleName, filter);
await client.AddRuleAsync(rule);
// Repeat this for a second rule with MainID = 101 and SecondID = 201
```
#### Expected behavior
We were expecting a logical AND behavior between filter properties, but a logical OR behavior between the rules (per subscription, that is).
#### Actual behavior
No messages are being picked up by the function app, because there would, of course, never be a situation where an incoming message would have `MainID = 100` **AND** `MainID = 101`. This seems to be caused by the rules following a logical AND behavior, similarly to a filter's properties.
#### Known workarounds
Atm. we let all messages emitted by the main system reach the function apps, only filtered by payload type. In the function apps we query out a row from a cloud table during the initial `Run`-method, wherein we already store some information (primarily temporary access tokens for the external system's API), and check if the `MainID` and `SecondID` is present (we mask the two IDs as the `TableEntity`'s `PartitionKey` and `RowKey`). This means, however, that the function app is spinning, costing us money, where simple filtering would've limited this. Unless you of course tell me know, that this is intended, because there are no cost benefits from filtering the messages down to being with... or that we're approaching this the wrong way...
#### Related information
* Programming language used: .NET Core 2.0
Contributor guide
Research direction
Start with the RuleDescription, CorrelationFilter, and AddRuleAsync behavior shown in the reproduction, then verify how multiple rules on one topic subscription are combined. Reproduce the two-rule case with the listed MainID, SecondID, and PayloadType values; done means the supported AND/OR semantics are established and the requested subscription behavior is either implemented or clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- backend, cloud, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100