Cluster takes 50GB memory for a 1B model

SKMPalettizer uses a C++ function called "cluster_impl". I have done some math and this function takes 40-50GB of memory to palletize a 1B model, which makes it almost impossible to run. In the comment it mentions that

""
// TODO: This step requires O(kn) memory usage due to saving the entire
// T matrix. However, it can be modified so that the memory usage is O(n).
// D and T would not need to be retained in full (D already doesn't need
// to be fully retained, although it currently is).
// Details are in section 3 of (Grønlund et al., 2017).
""

I wonder if this can be implemented?

Start with SKMPalettizer and its C++ cluster_impl function, then read the existing TODO about the O(kn) memory usage and section 3 of Grønlund et al. (2017). Trace how the D and T matrices are retained during palletization and verify that a 1B model no longer requires 40–50GB while preserving the clustering result.

Written by the indexing model from the issue text.