llnl / llnl/sundials

Segmentation fault in CVAfindIndex

Open
#31 2 comments 0 reactions 0 assignees View on GitHub
pkg-CVODES triage
Dominant language
C
Stars
686
Forks
167
Avg merge
1d 20h
Merged PRs (30d)
17

Description

I am using the adjoint sensitivity analysis functionality of sunodes, and sporadically I get segmentation faults during the backward pass. Unfortunately I could not reproduce this with a small example so far, but the coredump seems to indicate to me, that CVAfindIndex tries to access checkpoints that do not exist relatively near t0 of the forward problem, in a region where the solver is making (ridiculously?) small steps.
![Screenshot from 2020-02-26 11-18-52](https://user-images.githubusercontent.com/1882397/75335535-ee4cdd00-5889-11ea-8828-345fc105eeff.png)
It seems that CVAfindIndex is trying to find a checkpoint for `t = 161.33623519238427` but the largest of the 600 entries in ca_mem->dt_mem has only `t = 161.33623519238293`.

The details of how I'm using sundials are somewhat hidden in a python wrapper and pymc3 (I'm sampling the parameter space with an hamiltonian sampler), but here is a rough outline of what I'm doing:
- Initialize forward and backward solvers with polynomial interpolation and checkpoints every 600
- Repeat (a lot):
- Change user_data
- Call `CVodeReInit` and `CVodeAdjReInit`
- Run forward solver
- Call `CVodeReInitB`, `CVodeQuadReInitB` and `CVodeB`solve repeatedly, as the adjoint rhs is not continuous.

The `t` of the segfault is nowhere near the discontinuities of the rhs, the first one of those is at t ~ 12000.

The source for the solver calls is here: https://github.com/aseyboldt/sunode/blob/master/sunode/solver.py#L365

I can also provide the coredump if that is helpful.

Contributor guide

Open the contributing guide

Research direction

Start with CVAfindIndex and the solver calls in sunode/solver.py around line 365, then review the reported checkpoint times near t0. Try the described repeated CVodeReInit, CVodeAdjReInit, CVodeReInitB, CVodeQuadReInitB, and CVodeB sequence with 600 checkpoints; done means identifying and preventing the invalid checkpoint access, ideally with a reproducer or regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.