Azure / Azure/Azure-Sentinel

OCI_LogsV2_CL ingestion cost and duplicate columns

Open
#15,006 0 comments 0 reactions 3 assignees Claimed by @v-utpalkumar View on GitHub
Codeless Connector Framework (CCF) Connector Connector
Dominant language
Python
Stars
6.1k
Forks
3.8k
Avg merge
4d 7h
Merged PRs (30d)
125

Description

Describe the issue
The OCI_LogsV2_CL schema for the OCI CCF connector carries significant redundancy that inflates ingestion volume and storage cost. The raw data and oracle dynamic blobs are retained alongside the fully flattened data_* and oracle_* columns, so every field is stored twice, and several columns are exact duplicates of one another.

Evidence (measured on live data)
Raw-blob duplication, the data dynamic column re-contains the flattened values (0 differences across 6.5M rows):

tostring(data.identity.principalId) equals data_identity_principalId_s
tostring(data.compartmentId) equals data_compartmentId_s
tostring(data.eventName) equals data_eventName_s

Exact-duplicate column pairs (0 differences across 48M rows):

id equals id_s
TimeGenerated equals time_t
DstIpAddr equals data_destinationAddress_s
DstPortNumber equals data_destinationPort_d
DstBytes equals data_bytesOut_d
NetworkProtocol equals data_protocolName_s
SrcPortNumber equals data_sourcePort_d

Storing each field twice (raw blob plus flattened) is the dominant cost driver.

Proposed resolution (non-breaking, staged)
This is a GA connector, so columns cannot be removed in place without breaking existing customer queries, workbooks, and analytics. Proposed path:

Mark the redundant columns (the raw data and oracle blobs and the exact-duplicate pairs) as deprecated in the connector docs and release notes, with a stated removal window.
Provide a migration note pointing consumers to the retained canonical column for each deprecated one.
Remove them only after the notice window, in a clearly versioned release.

Additional context
Surfaced during the Cloud Guard work in PR #14859 (which was kept strictly additive). The redundancy analysis above is reproducible with the column-evidence KQL used during that PR review.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.