GoogleCloudPlatform / GoogleCloudPlatform/pubsec-declarative-toolkit

PubSub export queue of Org Logging Sink to 3rd party aggregators (Splunk HEC, IBM QRadar, ELK stack)

Open
#651 6 comments 0 reactions 1 assignee Claimed by @fmichaelobrien View on GitHub
dataflow logging pubsub splunk
Dominant language
Shell
Stars
36
Forks
26
PR merge metrics
No merged PRs in 30d

Description

# Note Log sink publishing delays - I see an hour+ on certain sinks - when using pubsub we are near realtime
thank you team for - https://support.google.com/a/answer/7061566

## Results
### Log Sink to Dataflow (apache beam) pipeline to Http Endpoint Collector

Screenshot 2023-11-17 at 16 03 49

## Notes
- check spec.disabled field found by Alain in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/680 around https://cloud.google.com/config-connector/docs/reference/resource-docs/logging/logginglogsink#custom_resource_definition_properties
- check https://cloud.google.com/security-command-center/docs/how-to-notifications
- UDF function to use with Splunk Dataflow template to transform events before delivery
- https://gist.github.com/rarsan/8d630bb59217d389542025508ff110bc
- https://github.com/GoogleCloudPlatform/terraform-splunk-log-export
- https://cloud.google.com/dataflow/docs/guides/templates/provided/pubsub-to-splunk
- 2021 https://cloud.google.com/blog/products/data-analytics/whats-new-splunk-dataflow-template
-

Screenshot 2023-11-09 at 1 37 32 PM

Screenshot 2023-11-09 at 1 37 01 PM

Screenshot 2023-11-09 at 1 34 35 PM

- splunk add on https://docs.splunk.com/Documentation/AddOns/released/GoogleCloud/About
- pubsub to splunk dataflow template https://cloud.google.com/dataflow/docs/guides/templates/provided-templates#streaming-templates

## Architecture/Requirements
- multiple log aggregators are pulling from GCP - in this case QRadar and Splunk - therefore a way to duplicate the logs or pubsub messages across 2+ clients is required.
- we can have 2 pubsub queues up with separate (duplicated) log sinks for each queue
- later we can look at a queue supplying to 2 queues for both aggregators if required
- FinOps should not be as affected if we limit the log sink retention as the source of truth will be the 3rd party aggregators not GCP

Support 3rd party log aggregation support for Splunk for example as a logging sync target

- https://cloud.google.com/architecture/stream-logs-from-google-cloud-to-splunk
- https://cloud.google.com/architecture/stream-logs-from-google-cloud-to-splunk/deployment

drive/shadow to https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/318

review splunk specific integrations

https://splunkbase.splunk.com/app/3088?_ga=2.266051020.2131231099.1699452385-302992155.1699452385&_gl=1*1oxfrkk*_ga*MzAyOTkyMTU1LjE2OTk0NTIzODU.*_ga_GS7YF8S63Y*MTY5OTQ1MjM4Ni4xLjAuMTY5OTQ1MjM4Ni42MC4wLjA.*_ga_5EPM2P39FV*MTY5OTQ1MjM4Ni4xLjEuMTY5OTQ1MjM4OS41Ny4wLjA.

review ELK stack export simulation

review IBM qradar integration
https://www.ibm.com/docs/en/qradar-common?topic=collector-setting-maximum-eps-rate

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.