DOI-USGS / DOI-USGS/streamMetabolizer
Two-station data prep: support real deployment data (gaps, multi-deployment, upstream covariates, per-day fitting)
- Dominant language
- Stan
- Stars
- 47
- Forks
- 25
- PR merge metrics
- No merged PRs in 30d
Description
The current two-station (VFTS) data prep pipeline (metab_bayes_2s(),
prepdata_bayes_2s(), mm_validate_data_2station()) was built to match the
shape of the reference dataset used in the original methodology - one
continuous deployment, regular timesteps, no gaps, downstream-only
covariates, always fit jointly. Several review comments on PR #469
surfaced real gaps between that assumption and what real-world monitoring
data actually looks like, plus one architectural question that reframes
several of them. Collecting these as one follow-up scope, since they're
facets of the same underlying question: what does two-station data prep
need to support real deployment data, not just the reference dataset?
Known facets (not a commitment to any particular solution yet):
- **User-facing lag/shift step**: moving the upstream/downstream lag
application out of internal-only logic (currently inside
prepdata_bayes_2s(), reaching into the previous date's raw rows) into an
exported preprocessing helper. This turns out to be the key piece several
of the items below depend on - see next item.
- **Per-day fitting**: currently, two-station always fits every date
jointly in one Stan call, which was assumed to be a Stan-model
limitation. It isn't - it's a consequence of the lag-shift happening
inline. If the lag-shift were a separate preprocessing step (above), each
date's modeled rows would be self-contained, and per-day fitting (like
one-station's `bayes_1ply()`) should be possible. This would also likely be
a prerequisite for `pool_K600` support, which needs the same per-day/
hierarchical structure one-station already has.
- **Gap tolerance / interpolation**: two-station currently has no per-day
timestep-regularity check at all (one-station has one, though see #474
for a bug in it). The published methodology linearly interpolated
single-timestep gaps and dropped days with larger gaps; informal team
discussion suggested a 4-8 timestep (1-2hr) tolerance. Building this means
new interpolation logic in the shared mm_is_valid_day()/
mm_filter_valid_days() validation layer (currently strict accept-or-
reject-whole-day only, no data mutation) - not a two-station-only change.
- **Upstream/averaged covariates**: light/depth/temp.water are currently
hardcoded to downstream-only. Real setups may measure these upstream, or
the team may prefer averaging paired upstream/downstream values.
- **Multi-deployment / phase-shifted timesteps**: concatenated deployments
with internally-regular but mutually-misaligned timesteps (e.g. one
deployment logging at :01/:16/:31/:46, the next at :04/:19/:34/:49)
aren't currently distinguished from a real data gap.
Also worth folding into this pass, lower priority: a pre-existing
one-station quirk noticed while auditing failed-run handling -
bayes_allply()'s comment claims a failed run's mcmc object is preserved,
but the code path that would attach it is skipped on failure, so it isn't.
Not urgent, but worth reconciling if/when this area gets touched again.
These are related but distinct enough that they likely need scoping as
separate pieces of work once picked up, not one big PR. Filed to capture
the discussion and guide a future design pass, not to commit to a specific
approach now.
Contributor guide
Research direction
Start by reading metab_bayes_2s(), prepdata_bayes_2s(), and mm_validate_data_2station(), then compare their assumptions with bayes_1ply() and the shared mm_is_valid_day()/mm_filter_valid_days() validation layer. Review PR #469 and issue #474 for context. Done should be a settled design that separates the distinct facets into scoped, independently actionable pieces rather than one implementation PR.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100