Roadmap: Onboarding optimizers to Trace and creating a benchmark for generative optimizers

Trace provides a framework to program agent architectures (parameterized by code, prompts, etc.) that can be trained by generative optimizers that can optimize graphs. There're many LLM-based generative optimization algorithms and agent optimization algorithms proposed in the literature. In principle, many are compatible with the Trace setting since they can be extended to go beyond their original goal (of optimizing texts) and work on graph directly. If we can have reliable implementation of these optimizers in Trace, then we can

1. Fairly compare their performance for research purpose. This addresses the issues that many experimental results in the literature are not directly comparable from an optimization algorithm's perspective, since there're differences in agents and prompts. This will help new research in generative optimization make progress faster and help its reproducibility.
2. Provide a suite of readily useable tools for practitioners. If multiple optimizers can be used interchangeably, a system developer can quickly experiment with different techniques to improve the system. This would lower the barrier of using generative optimization techniques. Currently, except for using Trace, switching algorithms means switching frameworks.

To achieve this goal, we need

1. Reliable implementation of generative optimization algorithms. Currently we have 3 in Trace. They can be made more reliable and we can increase the options.
2. Benchmark to test generative optimization algorithms. This arises as a necessary mean to onboard and debug new optimizers. We can start by repurposing the existing datasets that have been used in the literature and create evaluation of learning agents from them. The creation of this benchmark will help understand the performance of different optimization algorithms in the literature and help the process of developing new ones.

Next steps:

1. Create a list of algorithms to be implemented.
2. Create a list of datasets to be used as tests.

Start by reviewing the three existing generative optimizers in Trace and the datasets already used in the literature. Define the algorithm list and dataset list requested in the next steps, then determine how existing datasets can evaluate learning agents. Done means reliable optimizer implementations and a usable benchmark for comparing them.