facebookresearch / facebookresearch/CodeGen

`fastBPE` fix path

Open
#86 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
777
Forks
144
PR merge metrics
No merged PRs in 30d

Description

I against assertion `AssertionError: failed to learn bpe on /media/Z/dungnm31/transcoder/cpp-java-python.monolingual.tok.shuf.50gb, command: /home/dungnm/CodeGen/fastBPE/fast learnbpe 50000 /media/Z/dungnm31/transcoder/cpp-java-python.monolingual.tok.shuf.50gb > /media/Z/dungnm31/transcoder/cpp-java-python.monolingual.codes`

It turn out the command itself was not right. The `fastBPE` path will be located at _"codegen_sources/model/tools/fastBPE/fast"_ according to `install_env.sh` instead of _"fastBPE/fast"_

Suggest file `codegen_sources/preprocessing/bpe_modes/fast_bpe_mode.py` change
```python
FAST = str(Path(__file__).parents[3].joinpath("fastBPE/fast"))
```
to
```python
FAST = str(Path(__file__).parents[3].joinpath("codegen_sources/model/tools/fastBPE/fast"))
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.