Design: modifiers for loglike, score, hessian

Specific case: Where and how do we add penalization terms to the loglike, score and hessian?

Another case that is already in use is calling a transform_params to separate parameters to be optimized from internal representation. transform_params are currently mainly used for internal reparameterization (inequality constraints with interior solution) and yet not much for imposing equality constraints. The former are only relevant in optimization, the latter are also relevant for inference and other results.

The general task is that we need a wrapper that modifies the loglike and others on entry and exit. It would be a typical case for a decorator, except we don't want to hardcode a decorator in there.

see also my message in
https://groups.google.com/d/msg/pystatsmodels/qWJHRtPidaI/g8jpOJ4i7gYJ

The main purpose is for optimization with the standard scipy optimizers, but we need it also for inference, e.g. calculating the sandwiches for cov_params.
Optimization that uses other optimization methods can add a new fit method, but that still leaves robust cov_params as open question.

possible designs

• use subclasses and Mixin, e.g. class PoissonPenalized(PenalizedMixin, Poisson)
  • (+) doesn't require changes to existing models
  • (-) needs a new class for every existing class
• add optimizer options
  • current discrete fit_regularized adds separate optimizer that includes penalty, specific return classes
  • weaker option: outsource optimizers (PR) and allow overwriting functions/methods that are used in optimization, e.g. the lambda functions used in the optimizers based on loglike, score and hessian
  • (-) does not provide access to penalties/reparameterization for inference.
• build it into all models
  • check in each method, loglike, score, hessian, whether a "modifier" should be applied, similar to some of the current trans_params
  • call overhead: just one additional if check in the methods
  • needs a way to define the option to use it. current internal flags is not "nice" and doesn't generalize. Use keyword options instead. For that we need **kwds to be used by optimizers.
  • (-) needs to replicate code in every method
  • possible: refactor current loglike to leading underscore and add new loglike as wrapper. (similar to scipy.stats.distributions)
• combinations of the above.

other possibilities

• "prefix" class with delegation, mod_m = PenalizedMEstimator(modpr, smpen.SCADSmoothed(0.1, c0=0.0001))
• problem It's difficult to add all the necessary methods of the model class to the the prefix class

No source file or test is named. Start by reading the existing loglike, score, hessian, transform_params, fit_regularized, and cov_params implementations, then review the linked design discussion and scipy optimizer integration. Done means an agreed design explains how modifiers support optimization and inference without leaving robust covariance handling unresolved.