argosopentech / argosopentech/argos-train
OpenNMT-py v3 support
- Dominant language
- Python
- Stars
- 158
- Forks
- 29
- PR merge metrics
- No merged PRs in 30d
Description
https://forum.opennmt.net/t/opennmt-py-v3-0-is-out/5077
> The vanilla transformer uses sinusoidal positional encoding (position_encoding = true). We recommend to use “maximum relative positions” encoding instead (max_relative_positions=20, position_encoding=false) which again has a small overhead.
> We kept the “fusedadam” (old legacy code) which provides the best performance in speed (compare to pytroch amp adam fp16, apex level O1/O2). We tested the new Adam(fused=true) released with pytorch 1.13 but it is way slower.
> Always use the highest batch size possible (to your GPU ram capacity) and use an update interval according to the “true bach size” you want. For instance, if your GPU can accept 8192 tokens, then if you use accum_count=12, you will have a true batch size of 98304 tokens.
> Adjust the bucket size to your CPU ram. Most of the time a bucket between 200K and 500K examples will be suitable. The highest your bucket size is, the less padding you will have since examples are sorted based on this bucket and batches yield from this bucket.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the linked OpenNMT-py v3 announcement, then inspect the repository's Python training scripts to find the current OpenNMT-py integration and version assumptions. Done means the training scripts support OpenNMT-py v3 and reflect the relevant configuration guidance from the announcement, with any applicable checks passing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100