Importer skipping timesteps

Open
#69 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
38/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
java, python
Domain
data

Research direction

Start by reproducing the Vortex Importer behavior with InputData.zip, the clipping shapefile, and the documented NetCDF options, then compare the imported timesteps with the source data. Check both the Jython and GUI paths, including the intermittent STAGE IV GRIB case and the consistent NetCDF skips. Done means every expected timestep is written without unintended timestamp shifts.

Written by the indexing model from the issue text.

Description

So I'm having this issue where Vortex Importer is skipping timesteps. When using multiple STAGE IV radar GRIB files, it happens randomly, where sometimes it will skip a timestep or two, and sometimes it will write them all correctly, depending on its mood.

The workaround for that problem is to simply rerun it multiple times until it fills out the DSS. However, I'm also getting an issue where it will consistently skip over certain timesteps, when writing from a NetCDF gridset. This happens both when running from Jython and from the GUI.
EDIT: the consistent skips were fixed by duplicating the time coord var to an aux coor var called valid_time, as described below

The NetCDF's are CF-compliant, and have no missing timesteps. I've opened them up and plotted the timesteps that are not going through Vortex properly, and everything looks fine.

To reproduce

Attached below is one NetCDF and one clipping shapefile.
InputData.zip

Run the importer with the following options. I'm using the latest release but the bug was also happening with an older version before upgrading.
image
image

The following timesteps are the ones consistently dropping out. The same happens using other clip shapefiles as well. I'm also getting this surprising behavior where the single hourly timestamp is converted to a start time (D part) 0.5 hr earlier and an end time (E part) 0.5 hr later, so I'm having to timeshift the resulting DSS. I suppose that was the designed behavior, it just seems odd to me.

[Timestamp('2020-10-16 17:30:00'),
 Timestamp('2020-10-16 18:30:00'),
 Timestamp('2020-10-20 15:30:00'),
 Timestamp('2020-10-20 16:30:00'),
 Timestamp('2020-10-21 23:30:00'),
 Timestamp('2020-10-22 00:30:00'),
 Timestamp('2020-10-23 03:30:00'),
 Timestamp('2020-10-23 04:30:00'),
 Timestamp('2020-10-23 18:30:00'),
 Timestamp('2020-10-23 19:30:00')]
Dominant language
Java
Stars
37
Forks
7
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from HydrologicEngineeringCenter/Vortex

All issues in HydrologicEngineeringCenter/Vortex

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.