don't use sum_elems to get a scalar in Node unit tests.
- Dominant language
- C++
- Stars
- 3.4k
- Forks
- 701
- PR merge metrics
- No merged PRs in 30d
Description
A common pattern in our tests is to compute some Tensor-valued function, use sum_elems on it to reduce it to a scalar and then backprop and compare the evaluated gradient to the gradient computed on forward using finite differences. sum_elems is a bad scalar reduction because you end up with dy/d(element_i)=1 for every element. The resulting symmetry of the gradient can hide some common bug patterns (e.g., I found one hidden by this in fold_rows and circular convolution). A bit more reliable would be to reshape into a vector and do pickneglogsoftmax on some element to compute the loss- this the elements of this gradient will not be 1, so it will hide fewer bugs.
Contributor guide
No contributing guide indexed for this repository
Research direction
Inspect the Node unit tests that compute scalar losses with sum_elems, then compare the existing finite-difference and backpropagation checks with the suggested reshape and pickneglogsoftmax approach. Run the affected Node unit tests and confirm that gradient checks still pass without relying on uniform element gradients.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, node.js
- Domain
- machine-learning, testing-qa
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100