TuringLang / TuringLang/DynamicPPL.jl

Known issues with VNT, part 3 -- multiindex-upon-multiindex can give wrong keys with ArrayLikeBlocks

Open
#1,208 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

data-structures enhancement
Dominant language
Julia
Stars
286
Forks
41
Avg merge
1d 11h
Merged PRs (30d)
34

Description

using DynamicPPL
using DynamicPPL: templated_setindex!!

struct A end
Base.size(::A) = (2,)

vnt = VarNamedTuple()
vnt = templated_setindex!!(vnt, A(), @varname(x[2:4][1:2]), randn(4))

keys(vnt)
# 1-element Vector{VarName}:
#  x[1:2]

Here, we are setting indices 2 and 3 of the parent array x. However, keys(vnt) reports [1:2].

The reason for this is because the index in the key is obtained from the ArrayLikeBlock that is used to store the sized thing. In turn, the ArrayLikeBlock's indices are obtained from the indices that were used to initially set it inside the array, which were 1:2 -- it has no way of knowing that there was a 2:4 that came before it.

If this was x[1:N][P:Q] this would actually work perfectly fine for any N,P,Q but this issue is a worst-case scenario where the first set of multiindices don't begin at 1.

The fact that the stored indices are wrong can also lead to weird results when merging two VNTs with data that is mislabelled like this.

Note that the ALBs are still stored in the correct position in the PartialArray (indices 2 and 3 are the actual thing that we stored -- the other two indices are junk data generated via undef, but are masked so will never be seen). It's just the keys that are weird.

julia> vnt.data.x.data
4-element Vector{DynamicPPL.VarNamedTuples.ArrayLikeBlock{A, Tuple{UnitRange{Int64}}, @NamedTuple{}, Tuple{Int64}}}:
 ArrayLikeBlock(A(), (4526817440:4526817488,), NamedTuple(), (4526817536,))
 ArrayLikeBlock(A(), (1:2,), NamedTuple(), (2,))
 ArrayLikeBlock(A(), (1:2,), NamedTuple(), (2,))
 ArrayLikeBlock(A(), (0:0,), NamedTuple(), (131330,))

(This is good: it means that if we store an array instead of an ALB, the keys will be correctly picked up, because they are in the right positions.)

vnt = VarNamedTuple()
vnt = templated_setindex!!(vnt, randn(2), @varname(x[2:4][1:2]), randn(4))

keys(vnt)
# 2-element Vector{VarName}:
#  x[2]
#  x[3]

I personally think this is niche enough that we shouldn't need to care about it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the example with templated_setindex!!, @varname(x[2:4][1:2]), and an ArrayLikeBlock, then inspect how ArrayLikeBlock indices and PartialArray positions are used to form keys. Done means keys(vnt) reports the actual stored positions, such as x[2] and x[3], rather than x[1:2], without disrupting the correctly positioned data.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.