tensorflow / tensorflow/probability

Trainable priors in Conv2DFlipout layer

Open
#1,337 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
4.4k
Forks
1.1k
PR merge metrics
No merged PRs in 30d

Description

Hi all,

I am relatively new to tfp, possibly I have a rather basic doubt. Default Conv2DFlipout layer looks like this

    tfp.layers.Convolution2DFlipout(
        .
        .
        kernel_posterior_fn=tfp_layers_util.default_mean_field_normal_fn(),
        kernel_posterior_tensor_fn=(lambda d: d.sample()),
        kernel_prior_fn=tfp.layers.default_multivariate_normal_fn,
        kernel_divergence_fn=(lambda q, p, ignore: kl_lib.kl_divergence(q, p)), 
        .
        .
        **kwargs
    )

Can someone tell how can I involve kernel and bias priors in training. My intention is to use Empirical Bayes, in very similar lines as given in this blog by the tfp team. I would wish to train my prior mean and keep the prior standard deviation constant.

The default multivariate_normal_fn looks like this,

tfp.layers.default_multivariate_normal_fn(dtype, shape, name, trainable, add_variable_fn)

It has trainable as one of the boolean argument, if I set it True, it throws following error. (I was under the impression that it takes remaining arguments itself as in usual case and overwrites the trainable argument)

    tfp.layers.Convolution2DFlipout(
        kernel_prior_fn = tfp_layers_util.default_multivariate_normal_fn(trainable=True)
    )


    TypeError: default_multivariate_normal_fn() missing 4 required positional arguments: 'dtype', 'shape', 'name', and 'add_variable_fn'

Can someone help me solve the issue?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the Conv2DFlipout layer API and the default_multivariate_normal_fn signature, then compare the linked probabilistic-layers blog example. The issue does not identify a file or test; done would require a decided, supported way to make prior means trainable while keeping prior standard deviations fixed, along with validation of that behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.