JuliaDiff / JuliaDiff/ChainRulesCore.jl

Aliased differentials and Inplace Accumulation

Open
#350 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug design inplace accumulation
Dominant language
Julia
Stars
267
Forks
66
PR merge metrics
No merged PRs in 30d

Description

If a pullback is something like dx->(dx, dx) for dx being a reference type, e.g. like the pullback for +(::Array, ::Array), then the two outputs are the same object.
They are aliased.
Even though they are actually differentials for (usually) distinct (nonaliased) primal values.
This causes inplace accumulation to act wrong.

See discussion here https://github.com/FluxML/Zygote.jl/pull/962#issuecomment-835884201
where @mcabbott was just adding inplace accumulation for getindex.

I am wondering if we need to require that if the primals are not aliased the differentials also need to not be aliased.
This extra copies is slow if not inplace accumulation is not actually done though.
we might want to make it configurable depending on the AD system.
(this related to the config needs for #68)

I think the ideal solution would be copy-on-write objects.
But doing that without language support is suffering.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked discussion in pull request #962, especially the getindex inplace-accumulation work and the relationship to configuration issue #68. Determine the intended aliasing semantics for differentials, whether copying should be configurable, and what behavior would demonstrate that inplace accumulation is correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
tooling
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.