cdds_store failed to pick up where it left off when extending datasets

Migrated Jira Issue CDDSO-65
Jira creation date 21/04/2022 15:24
-------
In #2476 the archiving step for the onm stream failed on the first attempt with the error```CRITICAL: System Error while running MASS command: "moo put /project/cdds_data/CMIP6/ScenarioMIP/HadGEM3-GC31-LL/ssp245/r2i1p1f3/round-3/output/onm/Omon/thetao/thetao_Omon_HadGEM3-GC31-LL_ssp245_r2i1p1f3_gn_210001-210012.nc /project/cdds_data/CMIP6/ScenarioMIP/HadGEM3-GC31-LL/ssp245/r2i1p1f3/round-3/output/onm/Omon/thetao/thetao_Omon_HadGEM3-GC31-LL_ssp245_r2i1p1f3_gn_205001-209912.nc /project/cdds_data/CMIP6/ScenarioMIP/HadGEM3-GC31-LL/ssp245/r2i1p1f3/round-3/output/onm/Omon/thetao/thetao_Omon_HadGEM3-GC31-LL_ssp245_r2i1p1f3_gn_202101-204912.nc moose:/adhoc/projects/cdds/production/CMIP6/ScenarioMIP/MOHC/HadGEM3-GC31-LL/ssp245/r2i1p1f3/Omon/thetao/gn/embargoed/v20211117".This could cause by a system outage or when the system times-out as a result of excessive load.Traceback (most recent call last): File "/home/h03/cdds/software/miniconda3/envs/cdds-2.1.1/lib/python3.6/site-packages/transfer/command_line.py", line 49, in main_store num_critical_issues = store_mip_output_data(args) File "/home/h03/cdds/software/miniconda3/envs/cdds-2.1.1/lib/python3.6/site-packages/transfer/store.py", line 88, in store_mip_output_data archive_files(mip_approved_variables, arguments.simulate) File "/home/h03/cdds/software/miniconda3/envs/cdds-2.1.1/lib/python3.6/site-packages/transfer/mass.py", line 466, in archive_files run_archiving_commands(var_dict, simulation) File "/home/h03/cdds/software/miniconda3/envs/cdds-2.1.1/lib/python3.6/site-packages/transfer/mass.py", line 438, in run_archiving_commands check_mass_location=False) File "/home/h03/cdds/software/miniconda3/envs/cdds-2.1.1/lib/python3.6/site-packages/hadsdk/mass.py", line 349, in mass_put stdout_str = run_mass_command(moo_cmd) File "/home/h03/cdds/software/miniconda3/envs/cdds-2.1.1/lib/python3.6/site-packages/hadsdk/mass.py", line 551, in run_mass_command raise MassError(system_error, command)hadsdk.mass_exception.MassError: System Error while running MASS command: "moo put /project/cdds_data/CMIP6/ScenarioMIP/HadGEM3-GC31-LL/ssp245/r2i1p1f3/round-3/output/onm/Omon/thetao/thetao_Omon_HadGEM3-GC31-LL_ssp245_r2i1p1f3_gn_210001-210012.nc /project/cdds_data/CMIP6/ScenarioMIP/HadGEM3-GC31-LL/ssp245/r2i1p1f3/round-3/output/onm/Omon/thetao/thetao_Omon_HadGEM3-GC31-LL_ssp245_r2i1p1f3_gn_205001-209912.nc /project/cdds_data/CMIP6/ScenarioMIP/HadGEM3-GC31-LL/ssp245/r2i1p1f3/round-3/output/onm/Omon/thetao/thetao_Omon_HadGEM3-GC31-LL_ssp245_r2i1p1f3_gn_202101-204912.nc moose:/adhoc/projects/cdds/production/CMIP6/ScenarioMIP/MOHC/HadGEM3-GC31-LL/ssp245/r2i1p1f3/Omon/thetao/gn/embargoed/v20211117".```On retriggering the task a separate failure was triggered;```transfer.command_line.main_store CRITICAL: can only concatenate list (not "NoneType") to listTraceback (most recent call last): File "/home/h03/cdds/software/miniconda3/envs/cdds-2.1.1/lib/python3.6/site-packages/transfer/command_line.py", line 49, in main_store num_critical_issues = store_mip_output_data(args) File "/home/h03/cdds/software/miniconda3/envs/cdds-2.1.1/lib/python3.6/site-packages/transfer/store.py", line 88, in store_mip_output_data archive_files(mip_approved_variables, arguments.simulate) File "/home/h03/cdds/software/miniconda3/envs/cdds-2.1.1/lib/python3.6/site-packages/transfer/mass.py", line 466, in archive_files run_archiving_commands(var_dict, simulation) File "/home/h03/cdds/software/miniconda3/envs/cdds-2.1.1/lib/python3.6/site-packages/transfer/mass.py", line 420, in run_archiving_commands ret_val = preproc_mass_cmd(var_dict, simulation) File "/home/h03/cdds/software/miniconda3/envs/cdds-2.1.1/lib/python3.6/site-packages/transfer/mass.py", line 372, in move_files_for_prepending_and_appending_cmd check_mass_location=True) File "/home/h03/cdds/software/miniconda3/envs/cdds-2.1.1/lib/python3.6/site-packages/hadsdk/mass.py", line 392, in mass_move moo_cmd = ['moo', 'mv'] + src_mass_files + [dest_mass_path]TypeError: can only concatenate list (not "NoneType") to list```and there were CRITICAL issues of the form```transfer.mass.check_stored_status CRITICAL: Unable to process Omon/tauvo due to invalid mass state: Unknown invalid state.```for each variable that had been archived.I think this is due to the directory structure being a little messed up and the check_stored_status code not being able to interpret the state of the data when it is embargoed after extending a dataset.
-------

Trace the retry path through transfer/mass.py, especially check_stored_status, move_files_for_prepending_and_appending_cmd, and run_archiving_commands, then follow calls from transfer/store.py and transfer/command_line.py. Reproduce an extended dataset archive after an initial failure and verify that retriggering handles the embargoed state without the NoneType error or invalid-state failures.

Written by the indexing model from the issue text.