argosopentech / argosopentech/argos-train

OpenNMT-py v3 support

Open
#23 2 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Python
Stars
158
Forks
29
PR merge metrics
No merged PRs in 30d

Description

https://forum.opennmt.net/t/opennmt-py-v3-0-is-out/5077

> The vanilla transformer uses sinusoidal positional encoding (position_encoding = true). We recommend to use “maximum relative positions” encoding instead (max_relative_positions=20, position_encoding=false) which again has a small overhead.

> We kept the “fusedadam” (old legacy code) which provides the best performance in speed (compare to pytroch amp adam fp16, apex level O1/O2). We tested the new Adam(fused=true) released with pytorch 1.13 but it is way slower.

> Always use the highest batch size possible (to your GPU ram capacity) and use an update interval according to the “true bach size” you want. For instance, if your GPU can accept 8192 tokens, then if you use accum_count=12, you will have a true batch size of 98304 tokens.

> Adjust the bucket size to your CPU ram. Most of the time a bucket between 200K and 500K examples will be suitable. The highest your bucket size is, the less padding you will have since examples are sorted based on this bucket and batches yield from this bucket.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the linked OpenNMT-py v3 announcement, then inspect the repository's Python training scripts to find the current OpenNMT-py integration and version assumptions. Done means the training scripts support OpenNMT-py v3 and reflect the relevant configuration guidance from the announcement, with any applicable checks passing.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.