Performance enhancements for `init_strategy` may lead to unexpected behavior.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.8k
- Forks
- 316
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 27
Description
There is some dedicated logic to enhance the performance for specific init_strategys. This logic requires that init_strategy is a partial or a function that returns a partial. E.g., I expected, maybe naively, that the following init strategy would work.
>>> import jax
>>> import numpyro
>>>
>>>
>>> def model():
... numpyro.sample("x", numpyro.distributions.Normal())
... numpyro.sample("y", numpyro.distributions.Normal())
>>>
>>>
>>> def init_and_get_auto_loc(init_strategy):
... guide = numpyro.infer.autoguide.AutoDiagonalNormal(model, init_loc_fn=init_strategy)
...
... svi = numpyro.infer.SVI(model, guide, numpyro.optim.Adam(0.1), numpyro.infer.Trace_ELBO())
... state = svi.init(jax.random.key(9))
... return svi.get_params(state)["auto_loc"]
>>>
>>>
>>> init_strategy = lambda site: 3.0 if site["name"] == "x" else 7.0
>>> init_and_get_auto_loc(init_strategy)
TypeError: <lambda>() missing 1 required positional argument: 'site'
But wrapping in a partial works.
>>> from functools import partial
>>>
>>> init_and_get_auto_loc(partial(init_strategy))
Array([3., 7.], dtype=float32)
I came across this while trying to write an init strategy where some sites were initialized by value but the remainder initialized to uniform although with a different radius than the default of 2. Is this the intended behavior?
The relevant logic is here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in numpyro/infer/util.py at the linked lines around 373-384 and 742-748, then reproduce the lambda and functools.partial examples through AutoDiagonalNormal and SVI. Determine whether callable init strategies should receive the site argument in both paths, and consider the behavior complete when the reported strategy works consistently without regressing the partial optimization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100