Azure / Azure/azure-functions-host

Multiple RuleDescriptions per subscription

Open
#4,078 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

When adding multiple `RuleDescription` to a topic subscription, all added rules, and their filters, use a logical AND behavior. Why is this the case? Wouldn't it make more sense to have logical AND between filter properties of a rule, and then a logical OR between the separate rules (per subscription)? Or at least be able to indicate what the desired behavior should be, AND/OR, between properties and especially rules?

We use a single topic, `topics.employees`, where we have many different subscriptions, e.g. `EmployeeCreated`. The problem with this is, that we have many different types of customers using our primary system, and each of our customers have very specific and unique use-cases/needs. To avoid having to integrate many smaller, unique use-cases for external systems into our primary system that only one or maybe two of our customers need, we want to be able to instead build single Azure Functions per external system, that our customers might be in need of. These external systems have so far primarily been portals for our customer's employees, and thus, we have the topic called `topics.employees` in which we have a multitude of subscriptions per external system - e.g.: `ExternalSystemA_EmployeeCreated`. We'd want to be able to add rules to this subscription in such a fashion that it'd allow for processing only emitted messages from the certain customers who in fact have a need for the message to be processed by `ExternalSystemA_EmployeeCreated`.

Continuing from the example above, we have a main system where we emit an `EmployeeChangeMessage`, which is sent to our `topics.employees` as a brokered message. To the message we append the following properties: `MainID`, `SecondID`, `PayloadType`.

* `MainID` is a unique ID for a chain (think Wallmart as a whole)
* `SecondID` is a specific department of that particular chain (think an individual, unique Wallmart store - we do this, as some of our customers have use-cases where only some of their departments are in need of the external system)
* The `PayloadType` is used to distinguish between the different types of payloads we send from the main system (other than `EmployeeChangeMessage`, we also emit messages such as `SettingsChanged` etc.)

Because we want the main system to be as loosely coupled as possible, we do _not_ want it to know of the specific needs of each department, e.g.: "only emit the `EmployeeChangedMessage`, if one of these external systems are enabled...", or "add this extra property because this particular department has a certain need for a specific external system to process the message".
Thus, we want the function apps to be able to distinguish between the messages sent to the topic by (what we thought would be possible) having multiple rules for each external system's subscription, enabling us to filter messages before they reach the function app. Such as, Customer A has Departments A, B and C. Only emitted `EmployeeCreated` messages coming from Department A and C, should reach the subscription `ExternalSystemA_EmployeeCreated`.

#### Investigative information

Using function app version 2

Because I'm not the boss of my company, and we make our living by offering software solutions to our customers, and the fact that we're a small company who can't afford others capitalizing on our work, I'll refrain from sharing any code insight at this point, unless you find it _extremely_ important - although I find it hard to see why, as this is more question of design than actual locating a bug in our code.

#### Repro steps

1. Delete the initial default rule
2. Add a rule that will match against e.g. `MainID = 100` **AND** `SecondID = 200` **AND** `PayloadType = EmployeeCreated`
3. Add a rule that will match against e.g. `MainID = 101` **AND** `SecondID = 201` **AND** `PayloadType = EmployeeCreated`

(All steps done on the same subscription, of course)

```csharp
var filterProperties = new Dictionary
{
{"MainID", "100" },
{"SecondID", "200" },
{"PayloadType", "EmployeeCreated"}
};

var filter = new CorrelationFilter();
foreach (var pair in filterProperties)
filter.Properties.Add(pair.Key, pair.Value);

var rule = new RuleDescription(ruleName, filter);
await client.AddRuleAsync(rule);

// Repeat this for a second rule with MainID = 101 and SecondID = 201
```

#### Expected behavior

We were expecting a logical AND behavior between filter properties, but a logical OR behavior between the rules (per subscription, that is).

#### Actual behavior

No messages are being picked up by the function app, because there would, of course, never be a situation where an incoming message would have `MainID = 100` **AND** `MainID = 101`. This seems to be caused by the rules following a logical AND behavior, similarly to a filter's properties.

#### Known workarounds

Atm. we let all messages emitted by the main system reach the function apps, only filtered by payload type. In the function apps we query out a row from a cloud table during the initial `Run`-method, wherein we already store some information (primarily temporary access tokens for the external system's API), and check if the `MainID` and `SecondID` is present (we mask the two IDs as the `TableEntity`'s `PartitionKey` and `RowKey`). This means, however, that the function app is spinning, costing us money, where simple filtering would've limited this. Unless you of course tell me know, that this is intended, because there are no cost benefits from filtering the messages down to being with... or that we're approaching this the wrong way...

#### Related information

* Programming language used: .NET Core 2.0

Contributor guide

Open the contributing guide

Research direction

Start with the RuleDescription, CorrelationFilter, and AddRuleAsync behavior shown in the reproduction, then verify how multiple rules on one topic subscription are combined. Reproduce the two-rule case with the listed MainID, SecondID, and PayloadType values; done means the supported AND/OR semantics are established and the requested subscription behavior is either implemented or clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
backend, cloud, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.