GoogleChrome / GoogleChrome/webstatus.dev

[ENHANCEMENT] Check if WPT metric exists for feature before trying to insert

Open
#1,890 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
254
Forks
62
Avg merge
1d 10h
Merged PRs (30d)
64

Description

When the WPT Consumer job was first created, we told it to look at the last 2 [years](https://github.com/GoogleChrome/webstatus.dev/blob/43a2f84e5916842b4b98f7705bede2538a229739/infra/ingestion/workflows.tf#L106) of runs every time the job. This was because the IDs were still being added and we wanted the job to backfill.

This leads to really long run times. Almost a full day.

This may contribute to the high spikes in traffic to WPT too.

---
Possible solutions

We should probably add a capstone table similar to what we do for the UMA export job.

https://github.com/GoogleChrome/webstatus.dev/blob/43a2f84e5916842b4b98f7705bede2538a229739/infra/storage/spanner/migrations/000008.sql#L60-L65

The primary key would be a combination of the Web feature ID and the WPT Run ID.

Alternatively, we could look at the merging logic:

https://github.com/GoogleChrome/webstatus.dev/blob/43a2f84e5916842b4b98f7705bede2538a229739/lib/gcpspanner/wpt_run_feature_metric.go#L296-L316

If we detect there are no changes, we could emit no mutation. That could save us an insert.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.