Method for identifying comparable components across runs

One of our methods of evaluating reliability will be to compare ICA components across random seeds. From this we can look at the impact of convergence on the results and consistency of classification for equivalent components. I'm trying to figure out how we should do this.

Here are some proposed steps with potential pros/cons:

0. (Prerequisite) Run tedana with two seeds.
1. Load ICA mixing matrix and ICA component table from each run. These will have the components sorted in the same order (descending Kappa, I believe).
2. Correlate mixing matrices across the two runs, resulting in an n_comps X n_comps correlation matrix.
3. For each row in correlation matrix, identify index of maximum correlation coefficient.
   • Under optimal circumstances, this index would have each column represented once, with no duplicates. In reality, that does not seem to happen (see the correlation matrix I've added below). As you can see below, the extremely high correlations (yellow squares) sort of disappear further down.
   • How do we resolve duplicates, where a given component's highest correlation from one run is with more than one component from the other run?
4. To compare between convergence and non-convergence, compare distributions of these maximum correlation coefficients from converged/converged run pairs to converged/didn't-converge pairs.
   • We'll get an n_comps array of correlation coefficients from each pair, so to compare across all runs we'll need to use the full distributions.
   • As with all comparisons of convergence, a problem we'll have to deal with is that convergence failure doesn't happen randomly. Some subjects fail a lot of the time, while others never fail.
5. To evaluate consistency of classification, we'll need some metric summarizing cross-run comparability of components. Then we can build a contingency table (see example below) for each pair of runs, and can look at the average of that across all runs, I think.
   • We still have the duplicates issue here.

Example correlation matrix from real data

Example confusion matrix

Note that I'm ignoring the duplicates issue described above. That means that 8 components in run2 are reflected 2-3 times below, and 10 components are not reflected at all.

No files, tests, or entry points are named. Start by reviewing the proposed run-comparison workflow and the example matrices, then clarify the component-matching method, duplicate handling, and convergence/classification metrics; done means the project has an agreed, reproducible analysis plan.