facebookresearch / facebookresearch/CodeGen
`fastBPE` fix path
- Dominant language
- Python
- Stars
- 777
- Forks
- 144
- PR merge metrics
- No merged PRs in 30d
Description
I against assertion `AssertionError: failed to learn bpe on /media/Z/dungnm31/transcoder/cpp-java-python.monolingual.tok.shuf.50gb, command: /home/dungnm/CodeGen/fastBPE/fast learnbpe 50000 /media/Z/dungnm31/transcoder/cpp-java-python.monolingual.tok.shuf.50gb > /media/Z/dungnm31/transcoder/cpp-java-python.monolingual.codes`
It turn out the command itself was not right. The `fastBPE` path will be located at _"codegen_sources/model/tools/fastBPE/fast"_ according to `install_env.sh` instead of _"fastBPE/fast"_
Suggest file `codegen_sources/preprocessing/bpe_modes/fast_bpe_mode.py` change
```python
FAST = str(Path(__file__).parents[3].joinpath("fastBPE/fast"))
```
to
```python
FAST = str(Path(__file__).parents[3].joinpath("codegen_sources/model/tools/fastBPE/fast"))
```
Contributor guide
Assessment
This issue has not been assessed yet.