Replace duplication of sparse matrix

PROBLEM

For our assembly process, we currently:

1. fill a LvArray::SparsityPattern ( LvArray::ArrayOfSets) and then,
2. assimilate that into a LvArray::CRSMatrix.
3. In the assembly kernel, the dense local matrices are then inserted into the LvArray::CRSMatrix,
4. Then a HypreMatrix (contains a HYPRE_IJMatrix and a HYPRE_ParCSRMatrix) is created/allocated and the values are copied from the LvArray::CRSMatrix to the HypreMatrix through a call to HYPRE_IJMatrixAddToValues2.

This results in a 2x increase in matrix storage/allocation to run a specific problem . We need to implement a way to avoid this.

Solution

The solutions are varied, but they all essentially need to create a device callable interface around HYPRE_ParCSRMatrix, or an interface that performs a shallow copy of pointers containing data in HYPRE_ParCSRMatrix.

Option 1a: Shallow Copy. GEOS allocates new matrix type (LvArray::SplitSparsityPattern, LvArray::SplitCRSMatrix)

• Create the LvArray::SparsityPattern as we are doing now.
• Create a sparsity structure (LvArray::SplitSparsityPattern) that reflects the DIAG/OFF_DIAG nature of the HYPRE_ParCSRMatrix and copy the data from LvArray::SparsityPattern into that.
• Free LvArray::SparsityPattern.
• Create new matrix LvArray::SplitCRSMatrix.
• Fill LvArray::SplitCRSMatrix
• Shallow copy LvArray::SplitCRSMatrix into HYPRE_ParCSRMatrix.

Option 1b: Fill LvArray::SplitSparsityPattern

• Create a sparsity structure (LvArray::SplitSparsityPattern) that reflects the DIAG/OFF_DIAG nature of the HYPRE_ParCSRMatrix and copy the data from LvArray::SparsityPattern into that.
• Fill LvArray::SplitSparsityPattern.
• Create new matrix LvArray::SplitCRSMatrix.
• Fill LvArray::SplitCRSMatrix
• Shallow copy LvArray::SplitCRSMatrix into HYPRE_ParCSRMatrix.

Option 2a: Hypre allocates matrix

• Create the LvArray::SparsityPattern as we are doing now.
• Allocate HYPRE_ParCSRMatrix.
• Delete LvArray::SparsityPattern
• Create Wrapper around HYPRE_ParCSRMatrix and fill.

Option 2b: Hypre allocates matrix, but in 2 steps

• Create the LvArray::SparsityPattern as we are doing now.
• Allocate sparsity components of HYPRE_ParCSRMatrix
• Delete LvArray::SparsityPattern
• Allocate HYPRE_ParCSRMatrix.
• Create Wrapper around HYPRE_ParCSRMatrix and fill.