clab / clab/dynet

Riemannian optimization

Open
#768 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
3.4k
Forks
701
PR merge metrics
No merged PRs in 30d

Description

It would be cool to be able to do riemannian optimization directly with dynet.

As a reminder, what riemannian optimization does is

1. Modify the gradient (à la natural gradient)
2. Projects the parameters on the manifold after the update (à la weight clipping)

Some recent papers using this technique in non-trivial ways:

- [Poincaré Embeddings for Learning Hierarchical Representations](https://arxiv.org/abs/1705.08039) (optimize word/graph embedding in the hyperbolic poincaré sphere)
- [Riemannian Optimization for Skip-Gram Negative Sampling](https://arxiv.org/abs/1704.08059) optimization of the SGNS objective on a low rank manifold

I'm not sure if this should be implemented as separate trainers or by simplifying the manifold for each parameter (with *euclidean* as default). Any suggestions are welcome

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files, tests, or entry points. Start by locating DyNet's existing trainer implementations and the parameter-update path, then compare whether trainer-level or per-parameter support fits the current design. Done should include a defined API, gradient modification, manifold projection after updates, and coverage for the proposed behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.