SciML / SciML/Optimization.jl

Choice of second-order AD and warnings

Open
#970 4 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
839
Forks
100
Avg merge
20h 43m
Merged PRs (30d)
38

Description

This issue is about the machinery for choosing backends and throwing warnings:

https://github.com/SciML/OptimizationBase.jl/blob/2ffab7e93197c1fc8d9ed6a39857e301a71a474e/src/adtypes.jl#L222-L236

https://github.com/SciML/OptimizationBase.jl/blob/2ffab7e93197c1fc8d9ed6a39857e301a71a474e/src/cache.jl#L45-L58

I think that this could be both optimized and simplified due to recent changes in DI.

Nowadays, DI.inner and DI.outer can also be called on backends which are not SecondOrder, they just act as the identity. Thus, you don't need to explicitly create a SecondOrder(adtype, adtype). Passing adtype alone will be equivalent in most cases, and faster in some because it can leverage custom Hessian implementations within a single backend (e.g. SecondOrder(AutoForwardDiff(), AutoForwardDiff()) cannot call ForwardDiff.hessian whereas AutoForwardDiff() can).
Furthermore, DI's hvp and hessian for AutoZygote() already use ForwardDiff-over-Zygote.

Here are my suggestions:

  • Simplify the generate_adtype logic and its variants to avoid creating SecondOrder objects altogether.
  • Throw a warning based on the modes DI.inner and DI.outer, e.g. when the inner backend is not a reverse mode backend. This can be checked with ADTypes.mode(DI.inner(adtype)) isa Union{ADTypes.ReverseMode,ADTypes.ForwardOrReverseMode}. Of course you also want to allow ForwardDiff so feel free to refine.
  • Document this behavior so that users are less confused by the warnings (see this Discourse thread).

What do you think @Vaibhavdixit02?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/adtypes.jl lines 222-236 and src/cache.jl lines 45-58, then read the linked DifferentiationInterface inner, outer, hvp, and hessian behavior. Compare the current generate_adtype variants with the proposed mode-based warning logic. Done means the backend selection is simplified, warnings reflect the selected modes, and the behavior is documented for users.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
documentation, tooling
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.