DOI-USGS / DOI-USGS/ISIS3

Jigsaw failing on very large network under 10.X.X versions

Open
#6,153 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
245
Forks
181
Avg merge
1d 22h
Merged PRs (30d)
15

Description

**ISIS version(s) affected**: 10.0.0 and 10.1.0_RC1

**Description**
Jigsaw fails with the following error on a very large Kaguya TC morning network (168085 images, 12064753 points and 46368306 measures):
```
message from 10.1.0_RC1
Group = Error
Class = "PROGRAMMER ERROR"
Code = 3
Message = "Could not solve bundle adjust"
File = BundleAdjust.cpp
Line = 1236
Program = jigsaw
End_Group

Group = Error
Class = "PROGRAMMER ERROR"
Code = 3
Message = "Distance has not been initialized, you must initialize it first
before comparing with another distance using [>]"
File = Distance.cpp
Line = 208
Program = jigsaw
End_Group
```
Iteration one has a sigma 0 similar to the successful version, but things start going bad and diverging at iteration 2 and at iteration 4:

```
starting iteration 4
CHOLMOD warning: matrix not positive definite. file: ../Supernodal/t_cholmod_super_numeric.c line: 911

solve failed!
aborting...Object = jigsaw
```
isis10.0.0 fails at iteration 2 with these messages (it has an incredibly high sigma0 for iteration 1:
```
starting iteration 2

CHOLMOD warning: matrix not positive definite. file: ../Supernodal/t_cholmod_super_numeric.c line: 911

solve failed!
Bundle Complete
```
It actually creates the bundleout files despite "failing". No other error messages are reported for this version.

Note that this same network successfully bundles, converges and updates via isis9.0.0.

**How to reproduce**
This network is very large and is run on astro's cluster. The successful runs under 9.0.0 take up to 2 days without error prop (which does not complete under that version) or updating images.

My data are under my scratch area at Kaguya_TC/Global/Morning/GlobalNetwork/ErrProp/. The files in the top level of the directory are for the 10.1.0_RC1 run. See proc.scr for the jigsaw command. Note that the job was sent to the cluster exclusively using an entire node (435G memory). The job apparently stopped after 9.5h and was using 184G of memory at the time. This is more memory than what successful runs via 9.0.0 typically use (165G and usually 2+ days to run depending on network connectivity).

See I1000/ for the 10.0.0 run.

If interested in the successful 9.0.0 run I'll chat someone for that location. There's over 7Tb of images for this global network so troubleshooting needs to happen via the astro OS's and scratch area where the data reside.

**Additional context**
I have run jigsaw isis10.1.0_RC1 on very small networks successfully and even somewhat moderate sized Kaguya networks (south pole with 16204, 582998 points and 4535183 measures).

I plan on sending the LROC NAC south pole released network to isis10.0.0 to see if that fails. It is a smaller network but still very large (18323 images, 1677811 points and 14638903 measures) and the use case for adding cholmod64 to the bundle. It might be easier to troubleshoot if you actually need to walk through the network to see what the issue is.

Contributor guide

Open the contributing guide

Research direction

Start with proc.scr to understand the jigsaw command and compare the failing 10.0.0 and 10.1.0_RC1 runs with the successful 9.0.0 run. Inspect BundleAdjust.cpp at line 1236 and Distance.cpp at line 208 alongside the CHOLMOD failure output. Done means identifying and correcting the cause of the large-network bundle failure, then confirming the network converges.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.