NVIDIA / NVIDIA/cloudai

Make trajectory cache opt-in per workload (cache_safe contract)

Open
#886 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
99
Forks
62
Avg merge
6d 12h
Merged PRs (30d)
17

Description

Problem

CloudAIGymEnv.get_cached_trajectory_result() (in src/cloudai/configurator/cloudai_gym.py) returns the previously recorded (reward, observation) whenever a step's action matches an earlier entry's action. The returned tuple is then written to trajectory.csv and returned to the caller as if the workload had been re-executed.

This is correct only if the workload is deterministic given the action. For a stochastic workload, a cached (reward, observation) is a single sample from a distribution; reusing it instead of re-executing silently biases the recorded trajectory and any downstream consumer (DSE analysis, offline training corpora, leaderboards).

There is currently no way for a workload to declare that caching is unsafe for it.

Proposed change

Add a workload-level cache_safe: bool = True declaration on TestDefinition. CloudAIGymEnv.get_cached_trajectory_result() returns None whenever self.test_run.test.test_definition.cache_safe is False, forcing re-execution.

  • Default remains True to preserve current behavior for the existing deterministic workloads.
  • Stochastic workloads override to False in their TestDefinition subclass (or in TOML).
  • No change to consumers of trajectory.csv: the cache becomes a property of the workload, not a property the consumer has to reason about.

Acceptance criteria

  • TestDefinition exposes cache_safe: bool = True.
  • get_cached_trajectory_result() returns None when cache_safe is False, regardless of trajectory contents.
  • Unit test: a TestDefinition with cache_safe=False re-executes on a duplicate action; cache_safe=True (default) returns the cached entry as today.
  • Documentation note next to cache_safe stating the determinism contract.

Out of scope

Changing any existing workload's cache_safe value. Each workload owner decides separately.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/cloudai/configurator/cloudai_gym.py and locate the TestDefinition declaration and existing tests for get_cached_trajectory_result(). Trace how duplicate actions are handled, then add coverage for both cache_safe=False and the default behavior. Done means unsafe workloads re-execute while safe workloads retain the current cached result, with the determinism contract documented next to the declaration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.