Generalise the data processing pipeline such that we can analyse data iteratively, i.e. a few months at a time such that it gets added to the same dataframe

- Investigate different formats for data storage, and sub frames in parquet, for merging these different files. e.g. HDF5, [Feather](https://arrow.apache.org/docs/python/feather.html)
- Can selectively load elements of parquet to keep memory requirements down.
- Could store parts in different files and then merge in numpy afterwards.
- Check if float/double for indices
- For recorder/site numbers - should be int/categorical rather than continuous. Be careful for plotting/cross-referencing purposes.