Implement Automated User Provisioning and Deprovisioning from Directory Services

# CRITICAL NOTE
The below was generated by Claude without much contact on MJ conventions or existing functionality so first step we need is to define a plan that first studies existing MJ stuff and schema instead of blindly following this.

Also the Interface shown below might be best handled as an abstract base class so we can package some common routines and patterns into that.

## Problem Statement

Organizations using MemberJunction need to manually manage user accounts and role assignments, which is time-consuming, error-prone, and creates security risks. When employees join, leave, or change roles, IT teams must remember to manually update MJ, leading to:

- **Security gaps**: Former employees retaining access after departure
- **Onboarding delays**: New employees waiting for manual account creation
- **Role drift**: User permissions not reflecting their current organizational role
- **Administrative overhead**: Manual synchronization between directory and MJ
- **Audit compliance issues**: Difficulty tracking user lifecycle changes

## Proposed Solution

Implement automated user lifecycle management that integrates MJ with enterprise directory services through a provider abstraction layer. The sync process will be packaged as an MJ Action that can be scheduled and executed through MemberJunction’s existing Action framework.

## Architecture Overview

### Provider Abstraction Layer

```typescript
interface IDirectoryProvider {
// Connection & Authentication
initialize(config: DirectoryConfig): Promise;
testConnection(): Promise;

// User Operations
getUsers(filter?: UserFilter): Promise;
getUser(userId: string): Promise;
getUserGroups(userId: string): Promise;

// Group Operations
getGroups(filter?: GroupFilter): Promise;
getGroupMembers(groupId: string): Promise;

// Webhook Support (optional per provider)
supportsWebhooks(): boolean;
configureWebhook(callbackUrl: string): Promise;
validateWebhookPayload(payload: any): boolean;
}
```

### MJ Action Implementation

The sync process will be implemented as one or more MJ Actions:

- **DirectorySyncAction** - Main synchronization action (scheduled)
- **DirectoryWebhookHandler** - Real-time event processor (triggered by webhooks)
- **DirectoryManualSyncAction** - On-demand sync trigger (admin UI)

This leverages MJ’s existing Action scheduling infrastructure, allowing administrators to:

- Configure sync schedules via Action Parameters
- View sync execution history in Action logs
- Monitor sync performance through Action metrics
- Chain sync actions with other MJ Actions if needed

## Supported Directory Providers

### Phase 1 (Initial Release)

1. **SCIM 2.0 (Generic)** - Standards-based protocol
- Provides broad compatibility with most enterprise identity providers
- Supports: Okta, OneLogin, JumpCloud, Auth0, Azure AD, Google Workspace, and others
- Industry standard for user provisioning (RFC 7643, RFC 7644)
1. **Microsoft Entra ID (Azure AD)** - Native Microsoft Graph API implementation
- Enhanced features beyond SCIM (nested groups, advanced filtering)
- Better performance for Azure-centric organizations
- Delta queries for efficient incremental syncs
1. **Google Workspace** - Native Google Directory API implementation
- Enhanced features beyond SCIM (organizational units, advanced queries)
- Better performance for Google-centric organizations
- Push notifications for real-time updates

### Phase 2 (Future Consideration)

- **LDAP/Active Directory** - Direct LDAP protocol support for on-premises AD
- Additional provider-specific optimizations as needed

## Core Features

### 1. Auto-Provisioning (User Creation)

- Automatically create new MJ user accounts when users are added to the directory
- Map directory attributes to MJ user fields (email, name, department, etc.)
- Support filtering rules (e.g., only provision users from specific groups/OUs)
- Configurable default user settings and entity permissions
- Option for just-in-time (JIT) provisioning on first login vs. batch sync

### 2. Role Synchronization

- Map directory groups/roles to MJ roles automatically
- Bidirectional sync: directory → MJ roles and optionally MJ → directory
- Support for nested group membership
- Configurable mapping rules (e.g., “Azure AD Group: Developers” → “MJ Role: Data Analysts”)
- Handle role additions and removals dynamically
- Support for multiple role assignments per user

### 3. Auto-Deprovisioning (User Deactivation)

- Automatically deactivate MJ users when they’re disabled/deleted in the directory
- Soft delete by default (maintains audit trail and referential integrity)
- Configurable grace period before deactivation
- Notification system for deactivated users
- Orphaned user cleanup utilities

### 4. Synchronization Management via MJ Actions

**DirectorySyncAction Parameters:**

```yaml
Action Parameters:
- ProviderType: (SCIM | AzureAD | GoogleWorkspace | LDAP)
- ConnectionId: Reference to stored connection configuration
- SyncMode: (Full | Incremental | DryRun)
- SyncScope: (Users | Roles | Both)
- UserFilter: Optional filter expression
- ConflictResolution: (DirectoryWins | MJWins | ManualReview)
- NotifyOnCompletion: true/false
- NotificationRecipients: email addresses
```

**Scheduling Options:**

- Leverage MJ’s Action scheduling (cron expressions or interval-based)
- Common patterns: hourly, daily, weekly
- Peak/off-peak scheduling support
- Conditional execution based on previous run status

**Webhook Integration:**

- Real-time event processing for supported providers
- Webhook endpoint: `/api/directory/webhook/{providerId}`
- Triggers DirectoryWebhookHandler Action
- Falls back to scheduled sync if webhooks fail

### 5. Monitoring & Auditing

- Detailed sync logs using MJ’s Action execution logging
- Success/failure metrics via Action dashboard
- Audit trail stored in DirectorySyncLog entity
- Sync status visible in standard Action monitoring UI
- Error notification through Action notification system
- Performance metrics (users synced, duration, API calls)

## Technical Implementation

### Database Schema

**New Entities:**

```sql
-- Directory Provider Configuration
CREATE TABLE DirectoryConnection (
ID uniqueidentifier PRIMARY KEY,
Name nvarchar(255) NOT NULL,
ProviderType nvarchar(50) NOT NULL, -- SCIM, AzureAD, GoogleWorkspace, LDAP
Configuration nvarchar(MAX) NOT NULL, -- Encrypted JSON config
IsActive bit DEFAULT 1,
LastSyncAt datetime,
CreatedAt datetime DEFAULT GETDATE(),
UpdatedAt datetime DEFAULT GETDATE()
);

-- Sync Execution History
CREATE TABLE DirectorySyncLog (
ID uniqueidentifier PRIMARY KEY,
ConnectionID uniqueidentifier FOREIGN KEY REFERENCES DirectoryConnection(ID),
ActionExecutionID uniqueidentifier, -- Link to MJ Action execution
SyncType nvarchar(50), -- Full, Incremental, DryRun
StartTime datetime NOT NULL,
EndTime datetime,
Status nvarchar(50), -- Running, Success, Failed, PartialSuccess
UsersCreated int DEFAULT 0,
UsersUpdated int DEFAULT 0,
UsersDeactivated int DEFAULT 0,
RoleChanges int DEFAULT 0,
ErrorCount int DEFAULT 0,
ErrorDetails nvarchar(MAX),
Summary nvarchar(MAX)
);

-- Group/Role Mapping Configuration
CREATE TABLE DirectoryRoleMapping (
ID uniqueidentifier PRIMARY KEY,
ConnectionID uniqueidentifier FOREIGN KEY REFERENCES DirectoryConnection(ID),
DirectoryGroupId nvarchar(255) NOT NULL,
DirectoryGroupName nvarchar(255),
MJRoleID int FOREIGN KEY REFERENCES Role(ID),
IsActive bit DEFAULT 1,
CreatedAt datetime DEFAULT GETDATE()
);

-- Attribute Mapping Configuration
CREATE TABLE DirectoryAttributeMapping (
ID uniqueidentifier PRIMARY KEY,
ConnectionID uniqueidentifier FOREIGN KEY REFERENCES DirectoryConnection(ID),
DirectoryAttribute nvarchar(255) NOT NULL,
MJUserField nvarchar(255) NOT NULL,
TransformRule nvarchar(MAX), -- Optional JSON transformation logic
IsActive bit DEFAULT 1
);
```

**User Entity Extensions:**

```sql
ALTER TABLE User ADD ExternalDirectoryId nvarchar(255);
ALTER TABLE User ADD DirectoryConnectionID uniqueidentifier
FOREIGN KEY REFERENCES DirectoryConnection(ID);
ALTER TABLE User ADD LastSyncedAt datetime;
ALTER TABLE User ADD ProvisioningSource nvarchar(50); -- Manual, AutoSync, JIT
```

### Provider Implementations

**1. SCIM 2.0 Provider**

- Implements standard SCIM 2.0 endpoints:
- `/Users` - User resource management
- `/Groups` - Group resource management
- `/ServiceProviderConfig` - Provider capabilities
- `/Schemas` - Resource schema definitions
- Supports SCIM filtering and pagination
- Handles both SCIM 1.1 and 2.0 compatibility

**2. Azure AD Provider**

- Microsoft Graph API integration
- OAuth 2.0 authentication (app registration)
- Delta query support for incremental syncs
- Nested group resolution
- Change notification webhooks

**3. Google Workspace Provider**

- Google Directory API integration
- Service account authentication
- Organizational unit filtering
- Push notification support
- Advanced query capabilities

### Security Implementation

**Credential Storage:**

- All connection credentials encrypted at rest
- Use MJ’s existing encryption infrastructure
- Support for Azure Key Vault / AWS Secrets Manager integration
- Credential rotation support

**API Permissions (Minimum Required):**

- **Azure AD**: `User.Read.All`, `Group.Read.All`, `Directory.Read.All`
- **Google Workspace**: `https://www.googleapis.com/auth/admin.directory.user.readonly`, `https://www.googleapis.com/auth/admin.directory.group.readonly`
- **SCIM**: Provider-specific bearer token or OAuth client credentials

**Rate Limiting:**

- Respect provider API rate limits
- Implement exponential backoff
- Batch operations where supported
- Throttle configuration per provider

## User Interface Requirements

### Admin Configuration Panel

**1. Connection Setup Wizard**

- Provider selection (SCIM, Azure AD, Google Workspace)
- Connection credential entry (provider-specific)
- Connection test with detailed diagnostics
- Save as named connection

**2. Mapping Configuration**

- **Attribute Mapping**: Drag-and-drop or table-based UI
- Directory field → MJ User field
- Optional transformation rules
- Preview mapped values
- **Role Mapping**:
- List directory groups
- Map to MJ roles (many-to-many supported)
- Preview affected users

**3. Sync Schedule Configuration**

- Use MJ’s Action scheduling UI
- Configure DirectorySyncAction parameters
- Set up multiple scheduled syncs per connection
- Enable/disable schedules

**4. Monitoring Dashboard**

- Recent sync execution history (via Action logs)
- Success/failure statistics
- Last sync timestamp per connection
- Error alerts and notifications
- “Sync Now” manual trigger button

**5. Sync History Viewer**

- Detailed log of each sync execution
- Filterable by connection, date, status
- Drill-down into specific changes (users added/removed/updated)
- Export logs for compliance

### User Management Enhancements

- Badge showing provisioning source (Manual, Auto-Synced, JIT)
- Last sync timestamp display
- Link to source directory record
- Filter users by provisioning source and connection
- Bulk operations respect provisioning source (warnings for auto-provisioned users)

## Action Configuration Examples

### Example 1: Daily Full Sync

```yaml
Action: DirectorySyncAction
Schedule: "0 2 * * *" (2 AM daily)
Parameters:
ProviderType: AzureAD
ConnectionId: "azure-prod-connection"
SyncMode: Full
SyncScope: Both
ConflictResolution: DirectoryWins
NotifyOnCompletion: true
NotificationRecipients: ["admin@example.com"]
```

### Example 2: Hourly Incremental Sync

```yaml
Action: DirectorySyncAction
Schedule: "0 * * * *" (Every hour)
Parameters:
ProviderType: SCIM
ConnectionId: "okta-connection"
SyncMode: Incremental
SyncScope: Users
UserFilter: "department eq 'Engineering'"
ConflictResolution: DirectoryWins
```

### Example 3: Dry Run for Testing

```yaml
Action: DirectorySyncAction
Schedule: Manual
Parameters:
ProviderType: GoogleWorkspace
ConnectionId: "google-test-connection"
SyncMode: DryRun
SyncScope: Both
NotifyOnCompletion: true
```

## Acceptance Criteria

- [ ] Provider abstraction layer (`IDirectoryProvider`) is implemented and documented
- [ ] SCIM 2.0 provider is fully functional with Okta, OneLogin, or another SCIM provider
- [ ] Azure AD provider is fully functional with Microsoft Graph API
- [ ] Google Workspace provider is fully functional with Directory API
- [ ] DirectorySyncAction is registered and schedulable in MJ Action framework
- [ ] Auto-create new users when added to directory (within configured sync interval)
- [ ] Auto-deactivate users when disabled/deleted in directory
- [ ] Sync role memberships from directory groups to MJ roles
- [ ] Configurable attribute mappings work correctly for all providers
- [ ] Configurable role/group mappings work correctly
- [ ] Scheduled sync via MJ Actions runs reliably
- [ ] Manual sync trigger works from admin UI
- [ ] Webhook-based real-time sync works for Azure AD and Google Workspace
- [ ] Comprehensive sync logs are created and viewable in Action execution logs
- [ ] Error handling and notification system works through Action framework
- [ ] Dry-run mode allows previewing changes without applying them
- [ ] Connection test validates credentials and permissions
- [ ] Database schema changes are versioned and migrations provided
- [ ] Documentation covers setup for all three providers (SCIM, Azure AD, Google)
- [ ] Unit tests for provider abstraction layer (>90% coverage)
- [ ] Integration tests for each provider implementation (>80% coverage)

## Testing Scenarios

### Provider Compatibility Tests

1. **SCIM 2.0**: Test against Okta and OneLogin test tenants
1. **Azure AD**: Test against Azure AD test tenant
1. **Google Workspace**: Test against Google Workspace test domain

### Functional Tests

1. **New User Onboarding**: User added to directory → appears in MJ with correct roles
1. **User Departure**: User disabled in directory → deactivated in MJ
1. **Role Change**: User added to group in directory → role added in MJ
1. **Bulk Operations**: 100+ users added/removed → all sync correctly
1. **Conflict Resolution**: User manually modified in MJ → sync behaves per configuration
1. **Connection Failure**: Directory API unavailable → appropriate error handling and retry
1. **Rate Limiting**: High sync volume → respects API rate limits
1. **Incremental Sync**: Only changed users are processed in incremental mode
1. **Dry Run**: No actual changes applied, detailed preview generated
1. **Webhook Events**: Real-time events trigger immediate sync for affected users

### Action Framework Integration Tests

1. **Scheduled Execution**: Action runs at configured schedule
1. **Action Parameters**: All parameters correctly passed to sync logic
1. **Action Logging**: Sync results logged to Action execution log
1. **Action Chaining**: DirectorySyncAction can trigger other Actions (e.g., notifications)
1. **Error Handling**: Failed syncs properly reported in Action status

## Documentation Needs

### Administrator Documentation

- **Setup Guides** (per provider):
- SCIM 2.0: Generic configuration + provider-specific examples (Okta, OneLogin)
- Azure AD: App registration, permission grant, configuration
- Google Workspace: Service account setup, domain delegation, configuration
- **Mapping Configuration Guide**: How to configure attribute and role mappings
- **Scheduling Guide**: Setting up sync Actions with recommended patterns
- **Troubleshooting Guide**: Common issues and resolutions
- **Security Best Practices**: Credential management, permission scoping, audit requirements

### Developer Documentation

- **Provider Interface Documentation**: How to implement `IDirectoryProvider` for new providers
- **Action Development Guide**: Extending DirectorySyncAction with custom logic
- **Database Schema Documentation**: Entity relationships and field descriptions
- **API Reference**: Webhook endpoints and payload formats
- **Testing Guide**: How to test new provider implementations

### End User Documentation

- **What is Auto-Provisioning**: High-level overview for end users
- **Understanding Provisioning Status**: What the badges mean in user management
- **Troubleshooting Login Issues**: What to do if sync-related problems occur

## Open Questions

1. **Multi-Directory Users**: How do we handle users who exist in multiple directories?
- Proposed: Primary connection concept, with override capability
1. **Orphaned Users**: What happens to MJ users not found in any connected directory?
- Proposed: Configurable policy (ignore, flag, deactivate after grace period)
1. **Bidirectional Sync**: Should MJ be able to write back to directories?
- Proposed: Phase 2 feature, optional per provider
1. **Attribute Conflicts**: User manually updated in MJ, then auto-sync runs
- Proposed: Configurable conflict resolution (directory wins, MJ wins, manual review)
1. **Connection Priority**: Multiple connections, overlapping users
- Proposed: Explicit priority ordering on connections
1. **Just-In-Time Provisioning**: Auto-create on first login attempt?
- Proposed: Separate JIT provider, triggered during authentication
1. **Action Execution Concurrency**: Can multiple sync Actions run simultaneously?
- Proposed: Lock per connection to prevent concurrent syncs

## Implementation Phases

### Phase 1A: Foundation (Weeks 1-2)

- Database schema design and migration scripts
- Provider abstraction layer (`IDirectoryProvider`)
- Basic DirectorySyncAction framework
- Connection configuration storage (encrypted)

### Phase 1B: SCIM Provider (Weeks 3-4)

- SCIM 2.0 provider implementation
- User and group synchronization
- Testing with Okta/OneLogin
- Basic admin UI for connection setup

### Phase 1C: Azure AD Provider (Weeks 5-6)

- Microsoft Graph API integration
- OAuth authentication flow
- Delta query support
- Webhook integration

### Phase 1D: Google Workspace Provider (Weeks 7-8)

- Google Directory API integration
- Service account authentication
- Push notification support

### Phase 1E: Admin UI & Polish (Weeks 9-10)

- Complete admin configuration interface
- Mapping configuration UI
- Monitoring dashboard
- Documentation and testing

## Priority

**High** - This is a critical enterprise feature that reduces administrative overhead, improves security posture, and enables MemberJunction to be adopted at scale by organizations with existing identity infrastructure.

## Estimated Effort

**10-12 weeks** for Phase 1 with all three provider implementations (SCIM 2.0, Azure AD, Google Workspace) and full integration with MJ Action framework.

## Dependencies

- MJ Action framework must support parametrized Actions with scheduling
- Encryption infrastructure for secure credential storage
- Admin UI framework for configuration interfaces
- Entity metadata system for dynamic attribute mapping

Start by studying existing MemberJunction functionality and schema, especially the Action framework and current user and role entities; no implementation files are identified in the issue. Use the proposed DirectorySyncAction and provider abstraction as planning subjects, and check whether an abstract base class fits existing conventions. Done first means an agreed, scoped implementation plan identifying the initial provider, entities, and workflows before coding begins.

Written by the indexing model from the issue text.