GoogleCloudPlatform / GoogleCloudPlatform/pubsec-declarative-toolkit

FinOps: additional 150/month - 61GB security / 171GB other volume: Investigate excessive random 1ms frequency security log GKE ClusterManager.ListClusters audit calls causing spike from 0 to $5/day costs for the logging project

Open
#561 1 comment 0 reactions 1 assignee Claimed by @fmichaelobrien View on GitHub
FinOps
Dominant language
Shell
Stars
36
Forks
26
PR merge metrics
No merged PRs in 30d

Description

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/FinOps
fixing by turning off VPC flow logs
part of #492
61GB for the security log sink and 171GB for other logs
An idle autopilot cluster all of a sudden after a couple weeks starts incurring $5/day log storage costs with 1ms frequency GKE ClusterManager.ListClusters
This occurs on the logging project as a result of calls in the cluster project - log storage is not billed to the GKE cluster project
![image](https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/assets/24765473/6eb995a5-02ec-4075-a6c4-b7c3a19d8862)

![image](https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/assets/24765473/532e001e-813c-4e06-bd9e-9c7d6bd5ae6a)

![image](https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/assets/24765473/646bbc80-72a3-4100-8418-4bc1c64df956)

Review sink volume of 61GB on the security sink and 171GB on the other logs
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/client-landing-zone/logging-project/cloud-logging-bucket.yaml

Screenshot 2023-10-16 at 10 21 40

```
{
insertId: "1bg8ipsdcvuz"
logName: "projects/logging-project-kls/logs/cloudaudit.googleapis.com%2Fdata_access"
protoPayload: {
@type: "type.googleapis.com/google.cloud.audit.AuditLog"
authenticationInfo: {1}
authorizationInfo: [1]
methodName: "google.container.v1.ClusterManager.ListClusters"
policyViolationInfo: {1}
request: {2}
requestMetadata: {4}
resourceLocation: {1}
resourceName: "projects/logging-project-kls/zones/-"
serviceName: "container.googleapis.com"
}
receiveTimestamp: "2023-10-14T18:27:53.687821304Z"
resource: {2}
severity: "INFO"
timestamp: "2023-10-14T18:27:53.096002002Z"
}
```

triage deletion
```
root_@cloudshell:~ (kcc-kls-cluster3)$ gcloud logging logs delete security-log-bucket-kls --project=logging-project-kls
Really delete all log entries from [security-log-bucket-kls]?

Do you want to continue (Y/n)? y

ERROR: (gcloud.logging.logs.delete) NOT_FOUND: Log security-log-bucket-kls does not exist in the _Default bucket
root_@cloudshell:~ (kcc-kls-cluster3)$ gcloud logging logs list --project=logging-project-kls
NAME: projects/logging-project-kls/logs/cloudaudit.googleapis.com%2Factivity

NAME: projects/logging-project-kls/logs/cloudaudit.googleapis.com%2Fdata_access
root_@cloudshell:~ (kcc-kls-cluster3)$ gcloud logging logs list --project=logging-project-kls
NAME: projects/logging-project-kls/logs/cloudaudit.googleapis.com%2Factivity

NAME: projects/logging-project-kls/logs/cloudaudit.googleapis.com%2Fdata_access
root_@cloudshell:~ (kcc-kls-cluster3)$ gcloud beta logging logs list --project=logging-project-kls
NAME: projects/logging-project-kls/logs/cloudaudit.googleapis.com%2Factivity

NAME: projects/logging-project-kls/logs/cloudaudit.googleapis.com%2Fdata_access
root_@cloudshell:~ (kcc-kls-cluster3)$ gcloud beta logging logs delete --project=logging-project-kls
ERROR: (gcloud.beta.logging.logs.delete) argument LOG_NAME: Must be specified.
Usage: gcloud beta logging logs delete LOG_NAME [optional flags]
optional flags may be --help

For detailed information on this command and its flags, run:
gcloud beta logging logs delete --help
root_@cloudshell:~ (kcc-kls-cluster3)$ gcloud beta logging logs delete activity --project=logging-project-kls
Really delete all log entries from [activity]?

Do you want to continue (Y/n)? y

ERROR: (gcloud.beta.logging.logs.delete) NOT_FOUND: Log activity does not exist in the _Default bucket
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.