Proposal for feature extensibility

Expanding on OpenRiak/riak_test#2 I'm thinking a broader extension framework might be desirable so that we don't solve a single issue once then find we need more of the same and are constrained by the original solution.

Toward that end I propose something along the lines of the following:

%% license header
%% @doc Feature-specific data behavior for test consumption.
%%
%% By convention, tests are notified of the presence of a relevant data
%% callback in the `riak_test.config' file's `features' element:
%% ```
%%  {features, [callback_module1, ..., callback_moduleN]}
%% '''
%%
%% Implementing modules are to be stored in the `src' or `tests' directory to
%% which they are most relevant (almost always `tests', but future feature
%% extensions in the `riak_test' core are conceivable).
%%
%% A test can check for and possibly consume feature data in the following
%% manner:
%% ```
%%  AddlData = rt_feature:feature_data(some_data_class)
%% '''
%% Feature modules <i>MUST</i> export function `feature_data/1' whose final
%% function head shall be:
%% ```
%%  feature_data(_DataClass) ->
%%      undefined.
%% '''
%% The `DataClass' argument is deliberately flexible to allow parameterization
%% of the data retrieval function.
%%
%% @end
-module(rt_feature).

% Public API
-export([feature_data/1]).

-callback feature_data(DataClass :: term()) -> undefined | term().

-spec feature_data(DataClass :: term()) -> list(term()).
%% @doc Returns a list containing feature data from all defined feature
%% modules for the specified data class.
%% @returns A non-empty list of result data, or an empty list if no data
%% was present for the specified data class.
feature_data(DataClass) ->
    case rt:config(features) of
        [_|_] = FeatureModules ->
            lists:foldl(
                fun(FeatureModule, FeatureData) ->
                    case FeatureModule:feature_data(DataClass) of
                        undefined ->
                            FeatureData;
                        ModData ->
                            [ModData | FeatureData]
                    end
                end, [], FeatureModules);
        _ ->
            []
    end.

The use case example in OpenRiak/riak_test#2 would then be written as:

AllOrgStats = rt_feature:feature_data({org_admin_stats, all}),
OrgReplStats = rt_feature:feature_data({org_admin_stats, repl}),

There are, of course, details to be worked out regarding DataClass keys, but a sample implementing module for the above might be:

-module(myorg_features).
-behavior(rt_feature).

-export([feature_data/1]).

feature_data({org_admin_stats, all}) ->
    [this_stat, that_stat | feature_data({org_admin_stats, repl})];
feature_data({org_admin_stats, repl}) ->
    [myorg_repl_count];
feature_data(_) ->
    undefined.

This approach allows organizations to keep potentially non-public information only within their own repositories, while allowing data consumers such as verify_riak_stats and others that come along to be written with a constant pattern that can be oblivious to runtime configuration.

So, let's throw some darts at it!

Start by reviewing OpenRiak/riak_test#2 and the proposed rt_feature module, especially its use of rt:config(features) and feature_data/1. The issue leaves DataClass keys and other design details unresolved, so there is no agreed implementation boundary or concrete completion check yet.