Building ISCE3 0.4.0 from source code w/ CUDA capability

Hi, @vbrancat , @rtburns-jpl , and @gmgunter ,

Following the previous posting regarding ISCE3 conda package (Issue #7 ), here I would like to report the issue that I have experienced from building the software from source code (for CUDA capability).

I am using ISCE3 source code from public GitHub in branch "develop"

I have tried building the software on two linux-based systems as below:

Docker
Environment: Docker container from the image "nvidia/cuda:latest"
kernel: 5.10.47-linuxkit
gcc version 9.3.0
nvcc version 11.0.194
Host OS: MacOS ver 11.6 (Big Sur)
Docker version: 20.10.8, build 3967b7d

AWS EC2
kernel: 5.4.0-1045-aws
Instance type: p3.2xlarge
nvcc version: 11.5.50
gcc version 9.3.0
AMI: Ubuntu 20.04

What I did and saw:

• I installed the dependency using conda as in the instruction page below:
  
https://isce-framework.github.io/isce3/install_linux.html



• cmake error message has helped me to identify additional packages not listed on the instruction page.
  Few examples are eigen, pyre, gmock, etc.

• I had to use version-down few packages to make the building attempt successful.
  Few of them are hdf5 (to 1.10.6), pybind11 (to 2.6.2), and pyre (to 1.9.9).

• ctest: Bunch of tests were failed as below:

The following tests FAILED:
	126 - test.python.pybind..config (Failed)
	127 - test.python.pybind.core.attitude (Failed)
	128 - test.python.pybind.core.basis (Failed)
	129 - test.python.pybind.core.constants (Failed)
	130 - test.python.pybind.core.datetime_ (Failed)
	131 - test.python.pybind.core.ellipsoid (Failed)
	132 - test.python.pybind.core.euler (Failed)
	133 - test.python.pybind.core.interp1d (Failed)
	134 - test.python.pybind.core.kernels (Failed)
	135 - test.python.pybind.core.linspace (Failed)
	136 - test.python.pybind.core.LUT1d (Failed)
	137 - test.python.pybind.core.LUT2d (Failed)
	138 - test.python.pybind.core.orbit (Failed)
	139 - test.python.pybind.core.projections (Failed)
	140 - test.python.pybind.core.quaternion (Failed)
	141 - test.python.pybind.core.statevector (Failed)
	142 - test.python.pybind.core.timedelta (Failed)
	143 - test.python.pybind.focus.backproject (Failed)
	144 - test.python.pybind.focus.chirp (Failed)
	145 - test.python.pybind.focus.presum (Failed)
	146 - test.python.pybind.focus.rangecomp (Failed)
	147 - test.python.pybind.geocode.geocodeSlc (Failed)
	148 - test.python.pybind.geocode.geocodeCov (Failed)
	149 - test.python.pybind.geocode.radar_grid_cube (Failed)
	150 - test.python.pybind.geometry.bbox (Failed)
	151 - test.python.pybind.geometry.dem (Failed)
	152 - test.python.pybind.geometry.geo2rdr (Failed)
	153 - test.python.pybind.geometry.rdr2geo (Failed)
	154 - test.python.pybind.geometry.rtc (Failed)
	155 - test.python.pybind.geometry.metadata_cubes (Failed)
	156 - test.python.pybind.image.resamp (Failed)
	157 - test.python.pybind.io.gdal.dataset (Failed)
	158 - test.python.pybind.io.gdal.raster (Failed)
	159 - test.python.pybind.io.raster (Failed)
	160 - test.python.pybind.polsar.symmetrize (Failed)
	161 - test.python.pybind.signal.convolve2D (Failed)
	162 - test.python.pybind.signal.covariance (Failed)
	163 - test.python.pybind.signal.crossmul (Failed)
	164 - test.python.pybind.signal.crossmultiply (Failed)
	165 - test.python.pybind.signal.filter2D (Failed)
	166 - test.python.pybind.product.generic_product (Failed)
	167 - test.python.pybind.product.radargridparameters (Failed)
	168 - test.python.pybind.product.swath (Failed)
	169 - test.python.pybind.unwrap.icu (Failed)
	170 - test.python.pybind.unwrap.phass (Failed)
	171 - test.python.pybind.geometry.ltpcoordinates (Failed)
	172 - test.python.pybind.geometry.pntintersect (Failed)
	173 - test.python.pybind.antenna.frame (Failed)
	174 - test.python.pybind.antenna.geometryfunc (Failed)
	175 - test.python.pybind.cuda.core.compute_capability (Failed)
	176 - test.python.pybind.cuda.core.device (Failed)
	177 - test.python.pybind.cuda.focus.backproject (Failed)
	178 - test.python.pybind.cuda.geocode.geocode (Failed)
	179 - test.python.pybind.cuda.geometry.geo2rdr (Failed)
	180 - test.python.pybind.cuda.geometry.rdr2geo (Failed)
	181 - test.python.pybind.cuda.image.resamp (Failed)
	182 - test.python.pybind.cuda.signal.crossmul (Failed)
	183 - test.python.pkg.pybind_nisar.products.readers.attitude (Failed)
	184 - test.python.pkg.pybind_nisar.products.readers.orbit (Failed)
	185 - test.python.pkg.pybind_nisar.products.readers.raw (Failed)
	186 - test.python.pkg.pybind_nisar.workflows.crossmul (Failed)
	187 - test.python.pkg.pybind_nisar.workflows.focus (Failed)
	188 - test.python.pkg.pybind_nisar.workflows.stage_dem (Failed)
	189 - test.python.pkg.pybind_nisar.workflows.gcov (Failed)
	190 - test.python.pkg.pybind_nisar.workflows.geo2rdr (Failed)
	191 - test.python.pkg.pybind_nisar.workflows.gslc (Failed)
	192 - test.python.pkg.pybind_nisar.workflows.gpu_check (Failed)
	193 - test.python.pkg.pybind_nisar.workflows.geocode_insar (Failed)
	194 - test.python.pkg.pybind_nisar.workflows.insar (Failed)
	196 - test.python.pkg.pybind_nisar.workflows.point_target_info (Failed)
	197 - test.python.pkg.pybind_nisar.workflows.rdr2geo (Failed)
	198 - test.python.pkg.pybind_nisar.workflows.resample_slc (Failed)
	199 - test.python.pkg.pybind_nisar.workflows.rubbersheet (Failed)
	200 - test.python.pkg.pybind_nisar.workflows.unwrap (Failed)
	201 - test.python.pkg.pybind_nisar.workflows.antenna_parser (Failed)
	202 - test.python.pkg.pybind_nisar.workflows.dense_offsets (Failed)
	203 - test.python.pkg.pybind_nisar.workflows.cuda_insar (Failed)

But interestingly manually importing some of the failed modules are actually working on python. For example, the code below was working;

from isce3.io import Raster as raster # failed ctest no. 159

Also I was able to process some test data using CUDA.

So Two things I suspect:

• Building ISCE3 0.4.0 from source code is sensitive to the version of certain dependent packages
• Some changes in the object names have prevented ctest to run smoothly, even though the build was successful. For example, I noticed that the tutorial package imports raster from isce3.io, but it actually seems like Raster in this version of ISCE3.

Thanks again for the opportunity to try out this software.
Best,

Start with the Linux installation instructions and the listed ctest failures, then compare the dependency versions used in the Docker and AWS environments. Check the tutorial's raster import example against the installed Python API. Done means the source build has documented dependencies and the relevant ctest suite runs successfully.