tensorflow-cuda not working with greta
- Dominant language
- C++
- Stars
- 607
- Forks
- 67
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 1
Description
System:
- Arch linux 64bit
- Nvidia 960 (4gb)
- Nvidia driver: 410-66
- Cuda version: 10.0.130
- Using arch linux's tensorflow-opt-cuda (or tensorflow-cuda) for GPU support
Tested:
- GPU support is working. Used deviceQuery test for detection; GPU is supported
- Ran cifar10 example to benchmark GPU speeds; GPU support was working.
Greta model:
```
library(lavaan)
library(greta)
data(package='psych','bfi')
ds.a <- bfi[,1:5]
ds.a <- ds.a[complete.cases(ds.a),]
ds.a$A1 <- 7 - ds.a$A1
#ds.a <- ds.a[1:100,]
ds.a <- scale(ds.a)
ds.g <- as_data(ds.a)
N <- nrow(ds.a)
J <- ncol(ds.a)
# Latent CFA
theta <- normal(0,1,c(N,1))
nu <- normal(0,2,c(J,1))
lambda <- normal(0,1,c(J,1),truncation=c(0,Inf))
resid <- normal(0,2,c(J,1),truncation=c(0,Inf))
mu <- ones(N)%*%t(nu) + theta%*%t(lambda)
Sigma <- zeros(J,J)
diag(Sigma) <- resid
distribution(ds.g) <- multivariate_normal(mu,Sigma)
gretaMod <- model(lambda,nu,resid,theta)
gretaOut <- greta::mcmc(gretaMod,n_samples = 1000,warmup=1000,n_cores=1,chains = 1,one_by_one = FALSE)
```
I also tried with n_cores=4, chains=4; one_by_one=TRUE (I actually have to do this for this model due to some cholesky errors... which is odd, b/c that matrix is guaranteed positive definite, but I digress).
By default, compile=TRUE in model(). When that is TRUE, R/Rstudio will crash with a coredump very quickly. The error in the R session is: `tensorflow/compiler/xla/service/gpu/ir_emitter_unnested.cc:688] Check failed: fusion->fusion_kind() == HloInstruction::FusionKind::kLoop (kOutput vs. kLoop)`
No idea what that means, but on line 688 of that source file, there is indeed some equality check, and it fails.
If compile=FALSE, then the model seemingly runs fine (which makes sense, because XLA is not called).
compile=TRUE works fine when using CPU only; it appears to be a problem w/ gpu support.
Ironically, compile=TRUE w/ cpu only is much faster than compile=FALSE with gpu support. The GPU /is/ being utilized, but I suppose compilation greatly speeds things up.
Contributor guide
Assessment
This issue has not been assessed yet.