Some, not all, invalid ROIs are listed in `events` table of `BehaviorOphysExperiment` from S3 and LIMS

Describe the bug
When loading a BehaviorOphysExperiment from either LIMS or S3, the events table contains some invalid ROIs. It does not contain all invalid ROIs.

This issue seems intermittent and inconsistent. In the example I'll share below, for ophys_experiment_id 972233195, 17 of the 107 invalid ROIs are included in the events table. In another experiment, 974994093, the count is 1 of 99. In yet another, 907694954, this issue does not exist (there are 0 of 59 invalid ROIs included).

To Reproduce
Here is code to reproduce for experiment 972233195. This can be re-run for other experiments by subbing in a different oeid.

import allensdk.brain_observatory.behavior.behavior_project_cache as bpc
import mindscope_utilities.ophys

lims_cache = bpc.VisualBehaviorOphysProjectCache.from_lims()

data_storage_directory = "/allen/programs/braintv/workgroups/nc-ophys/visual_behavior/production_cache" # Note: this path must exist on your local drive
s3_cache = bpc.VisualBehaviorOphysProjectCache.from_s3_cache(cache_dir=data_storage_directory)

oeid = 972233195

lims_experiment = lims_cache.get_behavior_ophys_experiment(oeid)
s3_experiment = s3_cache.get_behavior_ophys_experiment(oeid)

Now we can query the length of the cell_specimen_table in LIMS:

len(lims_experiment.cell_specimen_table)

which returns 170

And the number of valid ROIs:

len(lims_experiment.cell_specimen_table.query('valid_roi'))

is 63

We'd expect the events table to include either all ROIs, or only valid ROIs. Instead:

len(lims_experiment.events)

tells us that it includes 80 entries

We can then get a list of the extra cell_specimen_ids as follows:

extra_lims_csids = list(set(lims_experiment.events.index) - set(lims_experiment.cell_specimen_table.query('valid_roi').index))
extra_lims_csids

which returns

[1086526338,
1086511716,
1086513222,
1086523174,
1086545258,
1086531340,
1086514893,
1086528300,
1086554255,
1086528079,
1086535645,
1086515411,
1086511252,
1086526234,
1086531836,
1086518621,
1086522591]

For the experiment from S3, the cell_specimen_table only lists the 63 valid ROIs:

len(s3_experiment.cell_specimen_table)

But the events table is 80 entries long:

len(s3_experiment.events)

We can then get the list of the 17 extra ROIs from S3:

extra_s3_csids = list(set(s3_experiment.events.index) - set(s3_experiment.cell_specimen_table.index))
extra_s3_csids

which returns the same list of cells, all of which are listed as valid_roi = False in the LIMS cell_specimen_table and do not exist in the S3 cell_specimen_table:

[1086526338,
1086511716,
1086513222,
1086523174,
1086545258,
1086531340,
1086514893,
1086528300,
1086554255,
1086528079,
1086535645,
1086515411,
1086511252,
1086526234,
1086531836,
1086518621,
1086522591]

Expected behavior
When loading from S3, I expect:

• The cell_specimen_table to list only valid ROIs (which is currently true)
• The events table to only include cells that are in the cell_specimen_table (which is not currently true)

When loading from LIMS, I expect:

• The cell_specimen_table to list all ROIs, with the valid_roi column denoting whether or not the ROI is valid (which is currently true)
• The events table to EITHER include only valid ROIs or all ROIs. I'm going to create a separate issue to disambiguate these two possibilities.

Actual Behavior
See above code block. The events table from both LIMS and S3 contains 17 invalid cells (in addition to 63 valid cells)

Environment (please complete the following information):

• OS & version: Ubuntu 18.04
• Python version 3.8
• AllenSDK version 2.11.2

Additional context
None

Do you want to work on this issue?
Yes

Start with VisualBehaviorOphysProjectCache.from_lims(), from_s3_cache(), and get_behavior_ophys_experiment() using the reproduction for experiment 972233195. Compare events with cell_specimen_table and its valid_roi values for both sources; done means the events table follows one documented, consistent ROI set without intermittent invalid entries.