missing data I/O and imputation / partial data structures

From Marcus:

Roughly speaking I was thinking of being able to write models like this;

data{
 matrix[N,M] Data;
}
transformed data {
  int missing_inds[];
  missing_inds <- find_missing_values(Data);
}
parameters {
  real missing_Data[length(missing_inds)];
}
transformed parameters{
  matrix[N,M] full_Data;

  full_Data <- fill_missing_values(Data,missing_inds,missing_Data);
}
model {
...
}

Here find_missing_values looks for NaN or something like that. If we didn't want to make it such a specific value, we call it something like find_nans so that users are clear in understanding what's going on here. Similar functions for the case where observed values are passed simply as index/value pairs would also be straightforward.

[We also need to] fix the dump file parsing to handle nan/inf.

My followup:

One attractive feature of this proposal is that we don't have to change the model for different models of missingness by column (as long as the constraints don't vary).

The one thing I was waffling back and forth on in my head was the issue of whether to take a full vector/matrix of predictors with NA-like placeholders or whether to have just the existing data passed in in "database" form:

  int n_rows;
  int n_cols;
  int n_observed;
  int observed_row[n_observed];
  int observed_column[n_observed];
  real observed_value[n_observed];

There are tighter data structures for anything but the sparsest matrices, but this representation seems easiest --- it's what you need for a "partial" matrix data structure.

Start by examining the dump file parsing path, which the issue specifically identifies as needing NaN/Inf handling. Before implementation, resolve whether missing values use full matrices with placeholders or observed row/column/value structures, then define completion criteria for the proposed missing-data functions and parsing behavior.