v2 Self-Tuning Budget for ResNet is Half of Benchmark Schedule-Free Run Time

V2 Self-Tuning Budget for ResNet is Half of Benchmark Schedule-Free Run Time

Description

I reran schedule-free adamw with the new self-tuning budgets to ensure that all workloads could reach the target in time. While most workloads fell within the budget, ResNet workloads required twice the new budget to reach the target.

ResNet Self-Tuning:

• v1 Budget (3*Old Ext): 189,024

• v2 Budget (1.5*New Ext): 99,238.5

• Schedule Free AdamW Runs:

  • Max: 189,245.496
  • Median: 181,588.748

• Proposed V2 Budget (3*New Ext): 198,477

Under the v1 ruleset, schedule-free ResNet workloads would reach the target at or slight after (200-300s) the budget. This budget was halved for the v2 ruleset when reduced from 3x to 1.5x the external-tuning budget.

As a result, ResNet workloads require 2x the new budget to reach the target. This seemed strange, considering that the self-tuning times were benchmarked with schedule-free adamw.

Steps to Reproduce

Run resnet workload with schedule-free adamw

Source or Possible Fix

Proposed ResNet Self-Tuning Budget: 198,477 (3*New Ext)

The external-tuning budget for ResNet was raised from 63,008 to 66,159 to reflect the longer-than-expected run times. It would make sense for the ResNet self-tuning budget to remain 3x the external-tuning budget and correspondingly increase from 189,024 to 198,477. As a result, schedule-free ResNet runtimes fall just within the new budget.

The issue names no files or tests, so first locate the ResNet self-tuning budget and schedule-free AdamW benchmark configuration. Run the ResNet workload to confirm the runtime, then update the budget to 198,477 and verify that the workload reaches its target within that budget.