clab / clab/dynet

Autobatch - Eigen Assertion Error

Open
#1,387 3 comments 0 reactions 0 assignees View on GitHub
moderate bug
Dominant language
C++
Stars
3.4k
Forks
701
PR merge metrics
No merged PRs in 30d

Description

This happens constantly, pretty much after 5 epochs of training a model. (or after 1 epoch after loading a model from disk)
I tried reinstalling DyNet and Eigen (from [here](http://dynet.readthedocs.io/en/latest/python.html#manual-installation)) but the error still persists.

> python: /home/nlp/amit/dynet-base/eigen/unsupported/Eigen/CXX11/../../../Eigen/src/Core/ProductEvaluators.h:161: static void Eigen::internal::Assignment, Eigen::internal::add_assign_op, Eigen::internal::Dense2Dense, typename Eigen::internal::enable_if<((Options == DefaultProduct) || (Options == AliasFreeProduct))>::type>::run(DstXprType&, const SrcXprType&, const Eigen::internal::add_assign_op&) [with DstXprType = Eigen::Map >; Lhs = Eigen::Map >; Rhs = Eigen::Transpose > >; int Options = 0; Scalar = float; Eigen::internal::Assignment, Eigen::internal::add_assign_op, Eigen::internal::Dense2Dense, typename Eigen::internal::enable_if<((Options == DefaultProduct) || (Options == AliasFreeProduct))>::type>::SrcXprType = Eigen::Product >, Eigen::Transpose > >, 0>]: Assertion `dst.rows() == src.rows() && dst.cols() == src.cols()' failed.

**More info:**
Running with CUDA does give an error at the same time, but a different error:
> CUBLAS failure in cublasSgemm(dev.cublas_handle, CUBLAS_OP_N, CUBLAS_OP_T, dEdxi.d.rows(), dEdxi.d.cols(), dEdf.d.cols() * dEdf.d.batch_elems(), dev.kSCALAR_ONE, dEdf.v, dEdf.d.rows(), xs[1]->v, xs[1]->d.rows(), dev.kSCALAR_ONE, dEdxi.v, dEdxi.d.rows())
7
terminate called after throwing an instance of 'dynet::cuda_exception'
what(): cublasSgemm(dev.cublas_handle, CUBLAS_OP_N, CUBLAS_OP_T, dEdxi.d.rows(), dEdxi.d.cols(), dEdf.d.cols() * dEdf.d.batch_elems(), dev.kSCALAR_ONE, dEdf.v, dEdf.d.rows(), xs[1]->v, xs[1]->d.rows(), dev.kSCALAR_ONE, dEdxi.v, dEdxi.d.rows())
Aborted (core dumped)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the failure after several training epochs and after loading a model from disk, then inspect the Eigen assertion in Eigen/src/Core/ProductEvaluators.h:161. Compare it with the reported cublasSgemm failure when running with CUDA; done means identifying the dimension mismatch and verifying a fix in both execution paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.