GoogleCloudPlatform / GoogleCloudPlatform/pubsec-declarative-toolkit
FinOps: additional 150/month - 61GB security / 171GB other volume: Investigate excessive random 1ms frequency security log GKE ClusterManager.ListClusters audit calls causing spike from 0 to $5/day costs for the logging project
- Dominant language
- Shell
- Stars
- 36
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/FinOps
fixing by turning off VPC flow logs
part of #492
61GB for the security log sink and 171GB for other logs
An idle autopilot cluster all of a sudden after a couple weeks starts incurring $5/day log storage costs with 1ms frequency GKE ClusterManager.ListClusters
This occurs on the logging project as a result of calls in the cluster project - log storage is not billed to the GKE cluster project



Review sink volume of 61GB on the security sink and 171GB on the other logs
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/client-landing-zone/logging-project/cloud-logging-bucket.yaml
```
{
insertId: "1bg8ipsdcvuz"
logName: "projects/logging-project-kls/logs/cloudaudit.googleapis.com%2Fdata_access"
protoPayload: {
@type: "type.googleapis.com/google.cloud.audit.AuditLog"
authenticationInfo: {1}
authorizationInfo: [1]
methodName: "google.container.v1.ClusterManager.ListClusters"
policyViolationInfo: {1}
request: {2}
requestMetadata: {4}
resourceLocation: {1}
resourceName: "projects/logging-project-kls/zones/-"
serviceName: "container.googleapis.com"
}
receiveTimestamp: "2023-10-14T18:27:53.687821304Z"
resource: {2}
severity: "INFO"
timestamp: "2023-10-14T18:27:53.096002002Z"
}
```
triage deletion
```
root_@cloudshell:~ (kcc-kls-cluster3)$ gcloud logging logs delete security-log-bucket-kls --project=logging-project-kls
Really delete all log entries from [security-log-bucket-kls]?
Do you want to continue (Y/n)? y
ERROR: (gcloud.logging.logs.delete) NOT_FOUND: Log security-log-bucket-kls does not exist in the _Default bucket
root_@cloudshell:~ (kcc-kls-cluster3)$ gcloud logging logs list --project=logging-project-kls
NAME: projects/logging-project-kls/logs/cloudaudit.googleapis.com%2Factivity
NAME: projects/logging-project-kls/logs/cloudaudit.googleapis.com%2Fdata_access
root_@cloudshell:~ (kcc-kls-cluster3)$ gcloud logging logs list --project=logging-project-kls
NAME: projects/logging-project-kls/logs/cloudaudit.googleapis.com%2Factivity
NAME: projects/logging-project-kls/logs/cloudaudit.googleapis.com%2Fdata_access
root_@cloudshell:~ (kcc-kls-cluster3)$ gcloud beta logging logs list --project=logging-project-kls
NAME: projects/logging-project-kls/logs/cloudaudit.googleapis.com%2Factivity
NAME: projects/logging-project-kls/logs/cloudaudit.googleapis.com%2Fdata_access
root_@cloudshell:~ (kcc-kls-cluster3)$ gcloud beta logging logs delete --project=logging-project-kls
ERROR: (gcloud.beta.logging.logs.delete) argument LOG_NAME: Must be specified.
Usage: gcloud beta logging logs delete LOG_NAME [optional flags]
optional flags may be --help
For detailed information on this command and its flags, run:
gcloud beta logging logs delete --help
root_@cloudshell:~ (kcc-kls-cluster3)$ gcloud beta logging logs delete activity --project=logging-project-kls
Really delete all log entries from [activity]?
Do you want to continue (Y/n)? y
ERROR: (gcloud.beta.logging.logs.delete) NOT_FOUND: Log activity does not exist in the _Default bucket
```
Contributor guide
Assessment
This issue has not been assessed yet.