Can we separate the generation of Q&A from the experiment process?

As part of an engagement, we are experimenting with the rag accelerator and find it extremely useful!

For evaluation, our plan is to leverage two sources of information:

1. Manually curated questions and answers that the SMEs created for us (around 15 pairs)
2. Generate additional questions and answers using LLMs

If I understand correctly, in 02_qa_generation, we generate these pairs and use those for evaluations. This process also generates the actual response from the LLM given the system prompt, user prompt and retrieved context.

My question is, how would you suggest to separate the two? If we generate q&a pairs every time, we might get different evaluation datasets and the experiments would not be comparable. In addition, we want to generate a q&a dataset and review it with our SMEs.

Start by reading the 02_qa_generation process to understand where question-answer generation and LLM responses are combined. Define the separation so a generated dataset can be saved, reviewed by SMEs, and reused for comparable experiments without regenerating it each time.