Composite `CeedOperator` collective context operations
- Dominant language
- C
- Stars
- 265
- Forks
- 78
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 11
Description
## Summary
Add an interface for collective operations over composite `CeedOperator` context values.
## Motivation
Currently, using the `CeedOperatorGetContext[Double|Int32|Boolean]Read` functions on a composite `CeedOperator` simply returns the value from the first sub-operator that has the context field defined. While useful for fields that should be the same for all sub-operators, e.g. time or timestep, there are many circumstances where each context would have a separate value which we want to perform a collective operation over. This is especially true for contexts which may have values written within a `CeedQFunction` (i.e. `CeedQFunctionSetContextWritable(ctx, true)`).
One place where this would be useful is in checking for element inversion, domain violation, or error checking, where a single boolean value context would be set to `true` within a `CeedQFunction` if an error is detected. Right now, in order to determine if any of the sub-operators of the composite operator `op` set the flag with name `name` in the context, a user must loop over the sub-operators and do something like the following:
```c
CeedInt num_sub;
CeedOperator *sub_operators;
bool value = false, is_set = false;
CeedOperatorCompositeGetNumSub(op, &num_sub);
CeedOperatorCompositeGetSubList(op, &sub_operators);
for (CeedInt i = 0; i < num_sub; i++) {
CeedContextFieldLabel label = NULL;
PetscSizeT num_values;
const bool *values_ceed;
CeedOperatorGetContextFieldLabel(sub_operators[i], name, &label);
if (label) {
CeedOperatorGetContextBooleanRead(sub_operators[i], label, &num_values, &values_ceed);
value = value || values_ceed[0];
is_set = true;
CeedOperatorRestoreContextBooleanRead(sub_operators[i], label, &values_ceed);
}
}
```
## Possible Interface
All of this is open to discussion and suggestion, but I think this would be a logical approach.
### Phase 1: Composite Operator Context Getter
Much of the user-side difficulty of the above approach can be alleviated by providing functions to get the values associated with a given field of each sub-operator of a composite `CeedOperator`. This interface would look like (using boolean fields as an example):
```c
CeedOperatorCompositeGetContextBooleanRead(CeedOperator composite_op, CeedContextFieldLabel label, CeedInt* num_suboperator_fields, size_t **num_values, const bool ***values_ceed);
CeedOperatorCompositeRestoreContextBooleanRead(CeedOperator composite_op, CeedContextFieldLabel label, CeedInt* num_suboperator_fields, size_t *num_values, const bool ***values_ceed);
```
where `*num_suboperator_fields` would be set to the number of suboperators with the requested field, `*num_values` would be set to the number of values in the field (already enforced to be constant across suboperators), and `*values` would be an array of pointers to each of the suboperator field value data (essentially the same as the current interface, just with an extra array over suboperators). That array could easily be cached if desired, though their size likely doesn't justify it.
Using this interface, the above code would be simplified to:
```c
CeedInt num_sub;
size_t num_values;
bool **values_ceed;
bool value = false, is_set = false;
CeedOperatorCompositeGetContextBooleanRead(op, label, &num_sub, &num_values, &values_ceed);
is_set = num_sub > 0;
for (CeedInt i = 0; i < num_sub; i++) value = value || values_ceed[i][0];
CeedOperatorCompositeRestoreContextBooleanRead(op, label, &num_sub, &num_values, &values_ceed);
```
### Phase 2: Collective Operation Utilities
I think that as a starting point, a generic backend function like:
```c
CeedOperatorContextCompositeReduceGeneric(CeedOperator op, CeedContextFieldLabel field_label, CeedContextFieldType field_type, CeedCollectiveOperation operation, size_t *num_values, void *values);
```
where `CeedCollectiveOperation` is an enum mirroring the default `MPI_Op` values (excluding `MINLOC` and `MAXLOC`):
```c
typedef enum {
CEED_OPERATION_MIN,
CEED_OPERATION_MAX,
CEED_OPERATION_BOR,
CEED_OPERATION_BXOR,
CEED_OPERATION_LOR,
CEED_OPERATION_LXOR,
CEED_OPERATION_BAND,
CEED_OPERATION_LAND,
CEED_OPERATION_SUM,
CEED_OPERATION_PROD,
} CeedCollectiveOperation;
```
Note, only some of the operations are valid for a given `CeedContextFieldType`. Specifically, [following MPI convention](https://www.mpi-forum.org/docs/mpi-1.1/mpi-11-html/node78.html):
```
[ MAX, MIN] CEED_CONTEXT_FIELD_DOUBLE, CEED_CONTEXT_FIELD_INT32
[ SUM, PROD] CEED_CONTEXT_FIELD_DOUBLE, CEED_CONTEXT_FIELD_INT32
[ LAND, LOR, LXOR] CEED_CONTEXT_FIELD_INT32, CEED_CONTEXT_FIELD_BOOL
[ BAND, BOR, BXOR] CEED_CONTEXT_FIELD_INT32
```
Then, the user-facing functions (note that the `values` array would be allocated by libCEED and freed by the user, or we could use a Get/Restore interface):
```c
CeedOperatorContextCompositeReduceBoolean(CeedOperator op, CeedContextFieldLabel field_label, CeedCollectiveOperation operation, size_t *num_values, bool **values);
CeedOperatorContextCompositeReduceDouble(CeedOperator op, CeedContextFieldLabel field_label, CeedCollectiveOperation operation, size_t *num_values, double **values);
CeedOperatorContextCompositeReduceInt32(CeedOperator op, CeedContextFieldLabel field_label, CeedCollectiveOperation operation, size_t *num_values, int32_t **values);
```
These functions should check that the requested operation is valid, then apply it over each of the `field_label->num_values` values and store the results in a new array, which is returned to the user as `*values`.
Contributor guide
Research direction
Start with the existing CeedOperatorGetContext[Double|Int32|Boolean]Read/Restore functions and the CeedOperatorCompositeGetNumSub and GetSubList entry points mentioned in the issue. First clarify whether Phase 1, Phase 2, or both are in scope with maintainers; done means an agreed public interface supports the specified composite context operations and validity rules.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- api, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100