Segmentation fault in CVAfindIndex
- Dominant language
- C
- Stars
- 686
- Forks
- 167
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 17
Description
I am using the adjoint sensitivity analysis functionality of sunodes, and sporadically I get segmentation faults during the backward pass. Unfortunately I could not reproduce this with a small example so far, but the coredump seems to indicate to me, that CVAfindIndex tries to access checkpoints that do not exist relatively near t0 of the forward problem, in a region where the solver is making (ridiculously?) small steps.

It seems that CVAfindIndex is trying to find a checkpoint for `t = 161.33623519238427` but the largest of the 600 entries in ca_mem->dt_mem has only `t = 161.33623519238293`.
The details of how I'm using sundials are somewhat hidden in a python wrapper and pymc3 (I'm sampling the parameter space with an hamiltonian sampler), but here is a rough outline of what I'm doing:
- Initialize forward and backward solvers with polynomial interpolation and checkpoints every 600
- Repeat (a lot):
- Change user_data
- Call `CVodeReInit` and `CVodeAdjReInit`
- Run forward solver
- Call `CVodeReInitB`, `CVodeQuadReInitB` and `CVodeB`solve repeatedly, as the adjoint rhs is not continuous.
The `t` of the segfault is nowhere near the discontinuities of the rhs, the first one of those is at t ~ 12000.
The source for the solver calls is here: https://github.com/aseyboldt/sunode/blob/master/sunode/solver.py#L365
I can also provide the coredump if that is helpful.
Contributor guide
Research direction
Start with CVAfindIndex and the solver calls in sunode/solver.py around line 365, then review the reported checkpoint times near t0. Try the described repeated CVodeReInit, CVodeAdjReInit, CVodeReInitB, CVodeQuadReInitB, and CVodeB sequence with 600 checkpoints; done means identifying and preventing the invalid checkpoint access, ideally with a reproducer or regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- hpc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100