GoogleCloudPlatform / GoogleCloudPlatform/pubsec-declarative-toolkit
PubSub export queue of Org Logging Sink to 3rd party aggregators (Splunk HEC, IBM QRadar, ELK stack)
- Dominant language
- Shell
- Stars
- 36
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
# Note Log sink publishing delays - I see an hour+ on certain sinks - when using pubsub we are near realtime
thank you team for - https://support.google.com/a/answer/7061566
## Results
### Log Sink to Dataflow (apache beam) pipeline to Http Endpoint Collector
## Notes
- check spec.disabled field found by Alain in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/680 around https://cloud.google.com/config-connector/docs/reference/resource-docs/logging/logginglogsink#custom_resource_definition_properties
- check https://cloud.google.com/security-command-center/docs/how-to-notifications
- UDF function to use with Splunk Dataflow template to transform events before delivery
- https://gist.github.com/rarsan/8d630bb59217d389542025508ff110bc
- https://github.com/GoogleCloudPlatform/terraform-splunk-log-export
- https://cloud.google.com/dataflow/docs/guides/templates/provided/pubsub-to-splunk
- 2021 https://cloud.google.com/blog/products/data-analytics/whats-new-splunk-dataflow-template
-
- splunk add on https://docs.splunk.com/Documentation/AddOns/released/GoogleCloud/About
- pubsub to splunk dataflow template https://cloud.google.com/dataflow/docs/guides/templates/provided-templates#streaming-templates
## Architecture/Requirements
- multiple log aggregators are pulling from GCP - in this case QRadar and Splunk - therefore a way to duplicate the logs or pubsub messages across 2+ clients is required.
- we can have 2 pubsub queues up with separate (duplicated) log sinks for each queue
- later we can look at a queue supplying to 2 queues for both aggregators if required
- FinOps should not be as affected if we limit the log sink retention as the source of truth will be the 3rd party aggregators not GCP
Support 3rd party log aggregation support for Splunk for example as a logging sync target
- https://cloud.google.com/architecture/stream-logs-from-google-cloud-to-splunk
- https://cloud.google.com/architecture/stream-logs-from-google-cloud-to-splunk/deployment
drive/shadow to https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/318
review splunk specific integrations
https://splunkbase.splunk.com/app/3088?_ga=2.266051020.2131231099.1699452385-302992155.1699452385&_gl=1*1oxfrkk*_ga*MzAyOTkyMTU1LjE2OTk0NTIzODU.*_ga_GS7YF8S63Y*MTY5OTQ1MjM4Ni4xLjAuMTY5OTQ1MjM4Ni42MC4wLjA.*_ga_5EPM2P39FV*MTY5OTQ1MjM4Ni4xLjEuMTY5OTQ1MjM4OS41Ny4wLjA.
review ELK stack export simulation
review IBM qradar integration
https://www.ibm.com/docs/en/qradar-common?topic=collector-setting-maximum-eps-rate
Contributor guide
Assessment
This issue has not been assessed yet.