Trajectory is O(N^2) per run in append and find; dominates runtime once trials are cheap

Summary

Trajectory is O(N²) in two independent places, both hit once per step. It is invisible when a trial costs tens of seconds, and it dominates completely once trials get cheap — which is the point of a surrogate/proxy model.

The two paths

append rebuilds the whole frame every step (src/cloudai/configurator/trajectory.py:128):

self._dataframe = lazy.pd.concat([self._dataframe, row_frame], ignore_index=True).astype(object)

A full copy plus an .astype(object) over every accumulated row, per append.

find scans with iterrows() every step (src/cloudai/configurator/trajectory.py:144-147). It is called from CloudAIGymEnv.get_cached_trajectory_result, which runs on every step to decide whether a configuration can be served from cache.

Measured

find alone, worst case (no match), pandas row loop over a 4-column frame:

Why it matters now

The cost only shows up relative to the trial cost:

• Real simulator, ~66 s/trial. 10,000 trials is ~183 hours. 30 minutes of bookkeeping is 0.3% — correctly ignorable, and presumably why this has never surfaced.
• Surrogate model, ~1 ms/trial. 10,000 trials is ~10 seconds of actual prediction against ~30 minutes of trajectory bookkeeping. The framework overhead is roughly 180x the work being done.

We hit the second case: a proxy model stands in for the simulator specifically so that DSE and RL runs can afford tens of thousands of trials. Making trials ~60,000x cheaper moved the bottleneck into Trajectory.

Suggested direction

Both are local to Trajectory and do not change its public surface:

1. find — maintain a dict keyed on the flattened (action, env_params) criteria, so a cache probe is a hash lookup rather than a scan. The method already computes exactly that key tuple in criteria.
2. append — accumulate rows in a list and materialise self._dataframe on demand (property, memoised, invalidated on append). That also removes the per-step .astype(object) over the full history.

The CSV write per append is fine and worth keeping — it is what makes a run recoverable, and it is what consumers read.

Happy to put up a PR if the direction is agreeable.

Found while reviewing a plugin repo's RL stack; measurements are a microbenchmark of iterrows at the relevant sizes, not an end-to-end profile.

Start in src/cloudai/configurator/trajectory.py at append (around line 128) and find (around lines 144-147), then trace CloudAIGymEnv.get_cached_trajectory_result to understand cache probes. Preserve the public surface, keep CSV recovery writes, and verify that repeated appends and cache lookups return the same results with substantially less per-step work.