OCI_LogsV2_CL ingestion cost and duplicate columns
- Dominant language
- Python
- Stars
- 6.1k
- Forks
- 3.8k
- Avg merge
- 4d 7h
- Merged PRs (30d)
- 125
Description
Describe the issue
The OCI_LogsV2_CL schema for the OCI CCF connector carries significant redundancy that inflates ingestion volume and storage cost. The raw data and oracle dynamic blobs are retained alongside the fully flattened data_* and oracle_* columns, so every field is stored twice, and several columns are exact duplicates of one another.
Evidence (measured on live data)
Raw-blob duplication, the data dynamic column re-contains the flattened values (0 differences across 6.5M rows):
tostring(data.identity.principalId) equals data_identity_principalId_s
tostring(data.compartmentId) equals data_compartmentId_s
tostring(data.eventName) equals data_eventName_s
Exact-duplicate column pairs (0 differences across 48M rows):
id equals id_s
TimeGenerated equals time_t
DstIpAddr equals data_destinationAddress_s
DstPortNumber equals data_destinationPort_d
DstBytes equals data_bytesOut_d
NetworkProtocol equals data_protocolName_s
SrcPortNumber equals data_sourcePort_d
Storing each field twice (raw blob plus flattened) is the dominant cost driver.
Proposed resolution (non-breaking, staged)
This is a GA connector, so columns cannot be removed in place without breaking existing customer queries, workbooks, and analytics. Proposed path:
Mark the redundant columns (the raw data and oracle blobs and the exact-duplicate pairs) as deprecated in the connector docs and release notes, with a stated removal window.
Provide a migration note pointing consumers to the retained canonical column for each deprecated one.
Remove them only after the notice window, in a clearly versioned release.
Additional context
Surfaced during the Cloud Guard work in PR #14859 (which was kept strictly additive). The redundancy analysis above is reproducible with the column-evidence KQL used during that PR review.
Contributor guide
Assessment
This issue has not been assessed yet.