JuliaDiff / JuliaDiff/ChainRulesCore.jl

Make inplace accumulation support Hermitian and Symmetric matrices

Open
#236 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

inplace accumulation
Dominant language
Julia
Stars
267
Forks
66
PR merge metrics
No merged PRs in 30d

Description

Follow up to https://github.com/JuliaDiff/ChainRulesCore.jl/pull/234

In that PR we are just saying that Hermitian and Symmetric matrices can not be used for in-place accumulation of gradients.
Conceptually though they really can; because pullbacks and pushforwards are linear operators, they preserve this structure: the differentials are also going to be Hermitian/Symmetric.

Its just that right now julia is really fussy about what operations you are allowed to do to them in such a way that its not easy to just treat them like any other matrix type.
https://github.com/JuliaLang/LinearAlgebra.jl/issues/773
In contrast we get away just fine treating (e.g.) Diagonal matrix's like any other matrix type (which is likewise safe because linear operator avoids doing anything illegal)

Anyway, we could do something to work around julia being fussy.
For example we could use parent to unwrap it to get the matrix behind it,
perform the inplace accumulation,
and then wrap it back up again.

This might mean we end up doing 2x as much work as we conceptually need to, since we only need to actually update one half.
and e.g. operations like parent(H1) .= H2 (where H1 and H2 are both Hermitian/Symmetric) will update both halves.
Similar for in-place multiplications.
It also feels pretty nasty; and means internally it will look like some functions violate the principle that the primal and the differential have the same structure. Which we might like to be asserting more and won't really be able to do as well with it internally manipulating the parent type.
But maybe that OK.

One might think that we could unwrap then rewrap as UpperTriangular(parent(H1)),
but that won't workout.
Even thought its much less fussy, and we don't run into these kinds of problems, it has has different invariants.
In this case that -- that you can't see lower triangular part to nonzero.
And we will end up doing operations like adding inplace a Symmetric/Hermitian matrix to it.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing PR #234 and JuliaLang/LinearAlgebra.jl issue #773, since no source files or tests are identified. Determine how in-place gradient accumulation handles Hermitian and Symmetric matrices and choose a way to support them without violating their invariants. Done means the supported accumulation paths work for both matrix types with appropriate coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.