Azure / Azure/Azure-Sentinel

Okta CCF connector: stuck poller re-ingests old events with new timestamps

Open
#14,984 0 comments 0 reactions 3 assignees Claimed by @hassanchawiche View on GitHub
Codeless Connector Framework (CCF) Connector Connector
Dominant language
Python
Stars
6.1k
Forks
3.8k
Avg merge
4d 7h
Merged PRs (30d)
125

Description

**Connector:** `Solutions/Okta Single Sign-On/Data Connectors/OktaNativePollerConnectorV2` (RestApiPoller)

## Summary

Under sustained event bursts, the connector gets stuck on a polling window it cannot complete and re-ingests that window's events repeatedly for days. Per `OktaSSOv2_PollingConfig.json`, the poller uses windowed `since`/`until` queries (`queryWindowInMin: 5`) with `LinkHeader` paging, `timeoutInSeconds: 60` and `retryCount: 3`. A burst window containing tens of thousands of events requires deep pagination that cannot finish within the timeout and retry budget (compounded by Okta's org-wide `/api/v1/logs` rate limit of 120/min shared with other consumers), so the window checkpoint never completes and the same pages are re-fetched and re-ingested on a cycle of roughly 12 to 17 hours. In our case this continued for more than 6 days.

Because Azure Monitor rewrites any incoming `TimeGenerated` older than about 2 days to ingestion time, these re-deliveries stop being harmless duplicates after 48 hours and become phantom events stamped with the current time, which then trigger scheduled analytics rules as if the activity just happened.

## Observed impact

- A single real `user.lifecycle.unsuspend` event (one uuid, verified against the Okta System Log API) was ingested 11 times over 6 days: 5 copies at the true `published` time (re-deliveries within 48h), then 6 copies with `TimeGenerated` rewritten to ingestion time on each subsequent re-poll. Each late copy fired a scheduled alert rule, and the "repeated admin actions" pattern triggered an internal security investigation into an admin who had acted exactly once.
- During replay cycles the table shows a surplus of 5k to 100k+ rows per hour versus the Okta API ground truth for the same window.
- The connector also drops events. In a sampled full day, 2 of 271 `group.user_membership.add` events returned by the Okta API never appeared in the table (both located inside the problematic burst window), alongside 27 duplicated events.

## Diagnosis recipe (for reproduction/verification)

1. Pick a high-volume window (e.g. an LDAP-interface event storm) and compare table row counts against `GET /api/v1/logs?since=...&until=...` with pagination, counting distinct `uuid`.
2. For any event uuid appearing more than once, compare `TimeGenerated` with `ingestion_time()`. Re-deliveries within 48h retain the mapped `published` time; later ones show `TimeGenerated` approximately equal to `ingestion_time()`. A secondary tell is timestamp precision: Okta-mapped times carry 3-digit millisecond precision, rewritten ones carry 7-digit ingestion precision.

## Suggested fixes

- Advance the checkpoint using Okta's `next` link cursor per page rather than per window, so a window that cannot complete does not cause a whole-window re-fetch.
- Deduplicate on event `uuid` before ingestion, or expose an idempotency option.
- Back off on 429s without resetting pagination state.
- At minimum, document that late re-deliveries older than 2 days will be re-stamped by the platform and can fire scheduled rules.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.