pyro-ppl / pyro-ppl/numpyro

Easier Use of Distributions Under `vmap`

Open
#2,271 3 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
2.8k
Forks
315
Avg merge
3d 9h
Merged PRs (30d)
27

Description

Distributions carry some static metadata (e.g., the batch shape), that make it a bit annoying to use in some other contexts. For example, they sometimes cause issues in dynestyx, where we use numpyro distributions quite generally.

An example of something I'd like to work, but doesn't, is just accessing variables of vmaped distributions:

def foo(mean):
    return dist.Normal(mean, jnp.ones_like(mean))

jax.vmap(foo)(jnp.zeros((2,2))).mean

Speaking offline with @fehiepsi, we agreed it would be desirable for such things to "just work." Looking back at previous discussions, however, it seems this is expected behavior under the current implementation (cf. https://github.com/pyro-ppl/numpyro/pull/1529#issuecomment-1411377292 or https://github.com/pyro-ppl/numpyro/issues/1684). Nevertheless, it would be nice to have a more extendable way of working with such things than imposing lots of numpyro-specific helpers on downstream users.

One way that might be possible is to try and defer shape evaluations when they may be inferred from parameter shapes (e.g., in all the scalar cases), in which case they become a lazy property. I had a quick vibe-coded go at this and it seems possible (see https://github.com/DanWaxman/numpyro/tree/dw-exploring-vmap-dists), though I haven't taken the time to go through that code very carefully (in particular, things that happen around expanded and transformed distributions), and some of the tests look wonky (in particular, I think Codex just changed test_distributions/test_promote_batch_shape_shares_data_and_preserves_input to pass ignoring all semantics haha), so please don't interpret it as a PR.

If that route seems interesting/acceptable, I'd be happy to take a closer/more careful look to see what it would take to do well and what it may break.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the distribution shape-handling code and tests referenced in the issue, especially test_distributions/test_promote_batch_shape_shares_data_and_preserves_input, alongside the dw-exploring-vmap-dists branch. Compare the vmap example with behavior around expanded and transformed distributions; done means the example works without breaking the cited shape and data-preservation semantics and the relevant tests are reliable.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.