Riemannian optimization
- Dominant language
- C++
- Stars
- 3.4k
- Forks
- 701
- PR merge metrics
- No merged PRs in 30d
Description
It would be cool to be able to do riemannian optimization directly with dynet.
As a reminder, what riemannian optimization does is
1. Modify the gradient (à la natural gradient)
2. Projects the parameters on the manifold after the update (à la weight clipping)
Some recent papers using this technique in non-trivial ways:
- [Poincaré Embeddings for Learning Hierarchical Representations](https://arxiv.org/abs/1705.08039) (optimize word/graph embedding in the hyperbolic poincaré sphere)
- [Riemannian Optimization for Skip-Gram Negative Sampling](https://arxiv.org/abs/1704.08059) optimization of the SGNS objective on a low rank manifold
I'm not sure if this should be implemented as separate trainers or by simplifying the manifold for each parameter (with *euclidean* as default). Any suggestions are welcome
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no files, tests, or entry points. Start by locating DyNet's existing trainer implementations and the parameter-update path, then compare whether trainer-level or per-parameter support fits the current design. Done should include a defined API, gradient modification, manifold projection after updates, and coverage for the proposed behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100