Adaptive Reminder Service - Proposal

# Adaptive Reminder Service for Orleans (related to #7573 )

> **Status:** Draft — open for feedback

---

## 1  Problem Statement

| Limitation | Impact in production |
|------------|---------------------|
| **Fixed‑interval only** | Cannot schedule “every weekday 09:00”, “last day of month”, etc. |
| **Partition‑wide scan** | A silo with **100 000+ reminders** loads them all into RAM at startup → long boot & GC churn (issues #947, #457). |
| **Missed tick is lost** | A cluster outage or GC pause during the due window silently drops that tick. |

---

## 2  Solution Overview

**Adaptive Reminder Service** replaces `LocalReminderService`.
It scales to hundreds of thousands of reminders, supports cron calendars, and guarantees at‑least‑once delivery while bounding CPU & memory.

### Highlights

* **Cron scheduling** — six‑field UTC cron expressions (second precision).
* **Adaptive buckets** — schedule only reminders due within a look‑ahead window; bucket size auto‑scales with CPU, memory and active‑grain count.
* **Three priorities** — `Critical`, `Normal`, `Background` let the runtime guarantee timeliness for critical work while deferring background tasks under load.
* **Per‑reminder missed‑tick action** — choose `FireImmediately`, `Skip` (default) or `Notify` (stream event); a background *repair job* fixes severely overdue rows.
* **Management grain** — `IReminderManagementGrain` lists, counts, edits and repairs reminders.
* **Startup helper** — `[RegisterReminder]` attribute recreates required reminders on silo boot if missing.
* **Backward compatible** — existing APIs compile unchanged; storage migration is additive.

---

## 3  Data Model

```csharp
public enum ReminderPriority : byte
{
Critical = 0,
Normal = 1,
Background = 2
}

public enum MissedReminderAction : byte
{
FireImmediately = 0,
Skip = 1,
Notify = 2
}

public sealed class ReminderEntry
{
// identifiers
public string ServiceId { get; init; }
public GrainId GrainId { get; init; }
public string ReminderName { get; init; }

// schedule
public DateTime StartUtc { get; init; }
public TimeSpan? Period { get; init; } // null ⇒ cron
public string? Cron { get; init; }

// runtime
public DateTime NextDueUtc { get; set; }
public DateTime? LastFireUtc { get; set; }
public ReminderPriority Priority { get; set; } = ReminderPriority.Normal;
public MissedReminderAction Action { get; set; } = MissedReminderAction.Skip;
}
```

> **Storage:** add columns `Cron`, `NextDueUtc`, `LastFireUtc`, `Priority`, `Action`; index `(NextDueUtc, Priority)`.

---

## 4  Configuration (`ReminderOptions`)

```csharp
public class ReminderOptions
{
public TimeSpan LookAheadWindow { get; set; } = TimeSpan.FromMinutes(5);
public int BaseBucketSize { get; set; } = 1_024;
public TimeSpan PollInterval { get; set; } = TimeSpan.FromSeconds(30);
public bool EnablePriority { get; set; } = true;
}
```

### Adaptive Bucket Formula

```
BucketSize = BaseBucketSize × CPU × MemoryFactor × GrainFactor

CPU = max(1, Environment.ProcessorCount / 4)
MemoryFactor = max(0.25, 1 − MemoryLoad%)
GrainFactor = min(1.0, 50_000 / ActiveGrainCount)
```

Example (16 cores, 60 % memory load, 40 000 grains) → `1 024 × 4 × 0.4 × 1 ≈ 1 640` concurrent timers.

---

## 5  API Additions

```csharp
public interface IReminderService
{
// interval‑based
Task RegisterOrUpdateReminder(
GrainReference grain, string name,
TimeSpan due, TimeSpan period,
ReminderPriority priority = ReminderPriority.Normal,
MissedReminderAction action = MissedReminderAction.Skip);

// cron‑based
Task RegisterOrUpdateReminder(
GrainReference grain, string name,
string cron,
ReminderPriority priority = ReminderPriority.Normal,
MissedReminderAction action = MissedReminderAction.Skip);
}
```
`IGrainReminder` now exposes `Cron`, `Priority`, `Action`.

---

## 6  Scheduling Algorithm

1. **Poll** storage every `PollInterval`; stream rows with `NextDueUtc ≤ now + LookAheadWindow`.
2. **Sort & bucket** by `Priority`, then `NextDueUtc`; bucket size uses adaptive formula.
3. **Queue** buckets to a `Channel`.
4. **Workers** consume channel under a `RateLimiter` (permits = CPU × 4).
5. **Execute** • overdue → apply `Action`; • otherwise delay until due, then deliver.
6. **Persist** new `NextDueUtc` & `LastFireUtc`.
7. **Repair job** (every minute) fixes rows with `NextDueUtc < now − LookAheadWindow`.

---

## 7  Management Grain

```csharp
public interface IReminderManagementGrain : IGrainWithGuidKey
{
Task> UpcomingAsync(TimeSpan horizon);
Task> ListForGrainAsync(GrainId grainId);
Task CountAllAsync();

Task SetPriorityAsync(GrainId grainId, string name, ReminderPriority p);
Task SetActionAsync (GrainId grainId, string name, MissedReminderAction a);
Task RepairAsync (GrainId grainId, string name);
Task DeleteAsync (GrainId grainId, string name);
}
```

---

### 8  _`[RegisterReminder]` Attribute_ **(BAD IDEA or not so bad 🙃)**

```csharp
[AttributeUsage(AttributeTargets.Class, AllowMultiple = true, Inherited = false)]
public sealed class RegisterReminderAttribute : Attribute
{
public string Name { get; }
public TimeSpan? Due { get; }
public TimeSpan? Period { get; }
public string? Cron { get; }
public ReminderPriority Priority { get; }
public MissedReminderAction Action { get; }

// interval‑based public RegisterReminderAttribute(
string name,
double dueSeconds,
double periodSeconds,
ReminderPriority priority = ReminderPriority.Normal,
MissedReminderAction action = MissedReminderAction.Skip)
{
Name = name;
Due = TimeSpan.FromSeconds(dueSeconds);
Period = TimeSpan.FromSeconds(periodSeconds);
Priority = priority;
Action = action;
}

// cron‑based
public RegisterReminderAttribute(
string name,
string cron,
ReminderPriority priority = ReminderPriority.Normal,
MissedReminderAction action = MissedReminderAction.Skip)
{
Name = name;
Cron = cron;
Priority = priority;
Action = action;
}
}
```

> At startup Orleans invokes `IReminderService.RegisterOrUpdateReminder` for each attribute — no temporary grain activation.

```csharp
[RegisterReminder("MonthlyReport", "0 0 0 1 * *", priority: ReminderPriority.Critical)]
[RegisterReminder("QuickPing", 30, 30)]
public sealed class StatsGrain : Grain, IRemindable
{
public Task ReceiveReminder(string name, TickStatus _) => Task.CompletedTask;
}
```

---

## 9  Integration Steps

1. **Schema upgrade** — add columns & index.
2. Register `AddAdaptiveReminderService()` in `ISiloBuilder`.
3. Update reminder providers (ADO, Azure Table, etc.) for new fields.
4. Document cron syntax, priority semantics, missed‑action matrix.

---

## 10  Open Questions

| Topic | Decision |
|-------|----------|
| Cron parser | Bundle NCrontab (MIT) vs. internal lightweight parser |
| Stream Id for `Notify` | Fixed GUID vs. configurable via `ReminderOptions` |
| Bucket constants | Fine‑tune via performance benchmarks |

Review the existing LocalReminderService and reminder provider entry points, then trace registration through ISiloBuilder and AddAdaptiveReminderService. Done would require an agreed design and implementation covering adaptive scheduling, cron support, storage changes, missed-tick handling, management APIs, and provider integration.

Written by the indexing model from the issue text.