tensorflow / tensorflow/probability
Issues with csiszar divergences
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 4.4k
- Forks
- 1.1k
- PR merge metrics
- No merged PRs in 30d
Description
I was doing some work of mine regarding the f-divergences, and was happy to find their implementations in TFP, however I feel like there're several issues at the moment.
-
The
tfp.vi.jensen_shannondivergence by default (withself_normalized=False) can be negative, since forf(u) = u log(u) - (1 + u) log(1 + u)f(1) = -2 ln(2). I don't see why one needsself_normalizedat all, sincef(u) = u log(u) - (1 + u) log(1 + u) + (u + 1) log(2)already satisfies bothf(u=1)=0andf'(u=1)=0. Same for thetfp.vi.arithmetic_geometric. Also, current implementation of the JSD is numerically unstable, but doesn't have to be. Here's a graph comparing currentf(logu)implementation against mine (with a control variate built in, see the next point):def _f_jsd_stable(x): lsx = tf.math.log_sigmoid(x) return tf.where(lsx != 0., tf.exp(x) * lsx, -tf.ones_like(x)) + lsx - x + 2. * tf.log(2.)

-
tfp.vi.total_variation,tfp.vi.triangularandtfp.vi.jensen_shannondivergences would benefit from a control variate of the forma (u-1). In particular, for the Total Variation one can use0.5 (u-1)as a control variate to obtainf(u) = max(0, 1-u), which is amazing since 1) it gives the same expectation, and thus defines the same divergence (for normalised measures); 2) this wayf(u) ∈ [0, 1]greatly reducing the variance; 3) all operations can be made numerically stable. Similar trick can be pulled with the Triangle divergence, leading to thef(u) = 4 / (1+u) - 2Other divergences do not seem to be amendable by such control variate, and I couldn't come up with any practical alternative. The only possibility is
c(u) = (a u + b) log uwhich can be expressedE_q(x) c(p(x)/q(x)) = a KL[p(x) || q(x)] - b KL[q(x) || p(x)], but this is only helpful for distributions with tractable KL, and is hard to implement in the current API.
I would be happy to contribute fixes
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the tfp.vi.jensen_shannon, tfp.vi.arithmetic_geometric, tfp.vi.total_variation, and tfp.vi.triangular divergence implementations. Compare their formulas with the proposed normalized and control-variate forms, then check numerical behavior using the _f_jsd_stable example. Done means the affected divergences are nonnegative where expected and numerically stable without changing their intended expectations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100