How to read evaluate output of a variational model

Hello,

I have been developing an VAE following some tutorials ( here , here, etc).

I am specially interested in studying the reconstruction error on out-of-distribution sets. Namely, I am implementing the loss function

negloglik = lambda x, rv_x: -rv_x.log_prob(x)

which is the reconstruction loss.

Furthermore, the encoding layer has a KL Divergence Regulator

tfpl.MultivariateNormalTriL(
        encoded_size,
        activity_regularizer=tfpl.KLDivergenceRegularizer(prior))

such that the wholve Loss is the ELBO (Evidence Lower Bound).

Now it comes my question, what is the best way of computing the reconstruction loss of a new sample? I have tried two approaches.

Approach 1: Sample from VAE model and pass it through reconstruction loss.

Consider we have X_test. I define the reconstruction preditor as var(X_test) such that my reconstruction losses are

losses_1 = - vae(X_test).log_prob(X_test).numpy().mean(1)

Approach 2: Use the evaluate method of VAE model.

Straightforwardly:

losses_2 = np.array([vae.evaluate(x.reshape(1,-1), x.reshape(1,-1), verbose=0) for x in X])

Findings

• Approach 2 is a lot slower to compute. While losses_1 takes around tens of milliseconds, approach losses_2 requires around 4 minutes for the same set. This is three orders of magnitude difference.
• losses_2 are 5 to 10 times higher than losses_1 for out-of-distribution sets, but overall similar for an in-distribution test set.

So, some questions:

• Is evaluate method also including the KL div loss?
• Why is evaluate so much slower?
• For out-of-distribution detection, should I use the evaluate method or just the reconstruction loss?
• Am I using the right methodology in approach 1? Or should I sample a reconstruction multiple times from the same sample?

Thanks for the help. Although I am quite versed in TF, Keras, DL, etc I am now learning about the power of Bayesian inference and it's applications.

Start with the issue's two entry points: vae(X_test).log_prob(X_test) and vae.evaluate(...), then inspect how KLDivergenceRegularizer(prior) contributes to the ELBO. Reproduce the timing and loss differences on in-distribution and out-of-distribution inputs; done means the issue documents whether KL is included, why the paths differ, and which reconstruction/OOD procedure is recommended.