Autobatch - Eigen Assertion Error
- Dominant language
- C++
- Stars
- 3.4k
- Forks
- 701
- PR merge metrics
- No merged PRs in 30d
Description
This happens constantly, pretty much after 5 epochs of training a model. (or after 1 epoch after loading a model from disk)
I tried reinstalling DyNet and Eigen (from [here](http://dynet.readthedocs.io/en/latest/python.html#manual-installation)) but the error still persists.
> python: /home/nlp/amit/dynet-base/eigen/unsupported/Eigen/CXX11/../../../Eigen/src/Core/ProductEvaluators.h:161: static void Eigen::internal::Assignment, Eigen::internal::add_assign_op, Eigen::internal::Dense2Dense, typename Eigen::internal::enable_if<((Options == DefaultProduct) || (Options == AliasFreeProduct))>::type>::run(DstXprType&, const SrcXprType&, const Eigen::internal::add_assign_op&) [with DstXprType = Eigen::Map >; Lhs = Eigen::Map >; Rhs = Eigen::Transpose > >; int Options = 0; Scalar = float; Eigen::internal::Assignment, Eigen::internal::add_assign_op, Eigen::internal::Dense2Dense, typename Eigen::internal::enable_if<((Options == DefaultProduct) || (Options == AliasFreeProduct))>::type>::SrcXprType = Eigen::Product >, Eigen::Transpose > >, 0>]: Assertion `dst.rows() == src.rows() && dst.cols() == src.cols()' failed.
**More info:**
Running with CUDA does give an error at the same time, but a different error:
> CUBLAS failure in cublasSgemm(dev.cublas_handle, CUBLAS_OP_N, CUBLAS_OP_T, dEdxi.d.rows(), dEdxi.d.cols(), dEdf.d.cols() * dEdf.d.batch_elems(), dev.kSCALAR_ONE, dEdf.v, dEdf.d.rows(), xs[1]->v, xs[1]->d.rows(), dev.kSCALAR_ONE, dEdxi.v, dEdxi.d.rows())
7
terminate called after throwing an instance of 'dynet::cuda_exception'
what(): cublasSgemm(dev.cublas_handle, CUBLAS_OP_N, CUBLAS_OP_T, dEdxi.d.rows(), dEdxi.d.cols(), dEdf.d.cols() * dEdf.d.batch_elems(), dev.kSCALAR_ONE, dEdf.v, dEdf.d.rows(), xs[1]->v, xs[1]->d.rows(), dev.kSCALAR_ONE, dEdxi.v, dEdxi.d.rows())
Aborted (core dumped)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the failure after several training epochs and after loading a model from disk, then inspect the Eigen assertion in Eigen/src/Core/ProductEvaluators.h:161. Compare it with the reported cublasSgemm failure when running with CUDA; done means identifying the dimension mismatch and verifying a fix in both execution paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100