Consider the function `combine_variable_lists(directory)`

This function was introduced here:
- #419

There are notes about why and how on the issue above.

Due to the way the tasks are parameterised, there is one "interim" variables file per recipe, and each dataset needs to retrieve the variables from all of them combined. Currently this is done by scanning a specified directory. Perhaps there is a better way?

Start with combine_variable_lists(directory) and read issue #419 for the existing rationale and parameterisation details. Trace how each dataset retrieves variables from the per-recipe interim files, then define and validate a better approach that still provides the combined variables without relying on scanning a specified directory.

Written by the indexing model from the issue text.