GoogleChrome / GoogleChrome/webstatus.dev
[ENHANCEMENT] Check if WPT metric exists for feature before trying to insert
- Dominant language
- Go
- Stars
- 254
- Forks
- 62
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 64
Description
When the WPT Consumer job was first created, we told it to look at the last 2 [years](https://github.com/GoogleChrome/webstatus.dev/blob/43a2f84e5916842b4b98f7705bede2538a229739/infra/ingestion/workflows.tf#L106) of runs every time the job. This was because the IDs were still being added and we wanted the job to backfill.
This leads to really long run times. Almost a full day.
This may contribute to the high spikes in traffic to WPT too.
---
Possible solutions
We should probably add a capstone table similar to what we do for the UMA export job.
https://github.com/GoogleChrome/webstatus.dev/blob/43a2f84e5916842b4b98f7705bede2538a229739/infra/storage/spanner/migrations/000008.sql#L60-L65
The primary key would be a combination of the Web feature ID and the WPT Run ID.
Alternatively, we could look at the merging logic:
https://github.com/GoogleChrome/webstatus.dev/blob/43a2f84e5916842b4b98f7705bede2538a229739/lib/gcpspanner/wpt_run_feature_metric.go#L296-L316
If we detect there are no changes, we could emit no mutation. That could save us an insert.
Contributor guide
Assessment
This issue has not been assessed yet.