happens-to-be-zero zero for tangent space of primal

Motivation

Somtimes it would be very convenient to have access to the element of the tangent space of a particular primal which happens to be zero for type-stability reasons. i.e. 0.0 and zeros(5, 4) rather than ZeroTangent.

The example I recently encountered where this would be helpful was reduce, specifically this line in Zygote's implementation of map. It's not possible to make this bit type-stable at the minute unless you know a priori whether or not the container you're mapping over is empty, so the pullback for map(function_with_fields, (5.0, 4.0)) infers, but the pullback map(function_with_fields, [5.0, 4.0]) does not. This is because the init kwarg is generally of a different type to the elements of Δf_and_args[1].

However, if we had access to a zero whose type doesn't change when we add cotangents to it, things ought to be type-stable.

Implementation

We know that the zero always exists, because the tangent space to a primal is a vector space, so there aren't any concerns regarding existence.

It's pretty clear what the right way to do this for composite types is via recursion (think rand_tangent, but zero rather than random), so we would just need to define it for primitives. It might get a little interesting here, because there are multiple possible tangent types for a Float64 primal (Float64, Float32, Float16, Int, etc) or a Vector primal (any AbstractVector of the same length with appropriate elements types), so possibly we would need additional information (such as the target tangent type) in order to do this.

Note that we do need to know the value of the primal for the same reasons that we need to have the primals hanging around in our projection functionality.

Anyway, I thought I'd bring this up because it's not something that we've thought much about before (ZeroTangent is often a really good option). It might be easier simply avoid situations like this most of the time (e.g. in the example I mentioned, using sized containers), but it's pretty annoying that the pullback for map isn't type-stable when mapping a closure over a Vector, because people to that a lot.

Start by reading the linked Zygote src/lib/array.jl line around reduce and map, then inspect ChainRulesCore's rand_tangent and projection functionality. The issue proposes a value-dependent zero tangent for primitive and composite types, but leaves the API and tangent-type selection unresolved; done would require an agreed design and implementation covering those cases.