facebookresearch / facebookresearch/CodeGen

Outputted files have erroneous characters [bug on mac os]

Open
#17 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
777
Forks
144
PR merge metrics
No merged PRs in 30d

Description

Hi, I was trying to reproduce the results in the TransCoder paper, but ran into some issues when it computed the computational accuracy.
It seems that when writing things such as id’s or outputted programs to files or outputs in the log, the text has some issues.

For example, in ids.java_sa-python_sa.text.txt (created by create_reference_files in evaluator.py), the lines look like “CHECK_@@ WHE@@ THER_@@ GI@@ V@@ EN_@@ NUMBER_@@ EV@@ EN_@@ O@@ DD”.

The scripts outputted by the model (e.g. in eval_scripts/java_sa-python_sa.test/) that are used to compute the computational accuracy similarly have erroneous characters and spaces (which causes syntax errors), e.g.:
```
def f_filled is_@@ ap ( arr , n ) :
if n == 1 : return True
arr.sort ( )
d = arr [ 1 ] - arr [ 0 ]
for i in range ( 2 , n ) :
if arr [ i ] - arr [ i - 1 ] != d : return False
return True
```
If it is relevant, I am on Mac OS Catalina with python 3.9 and this is the command I have been running to evaluate the TransCoder models provided:
```
python codegen_sources/model/train.py \
--eval_only True \
--reload_model 'TransCoder_model_1.pth,TransCoder_model_2.pth' \
--data_path "test_dataset" \
--exp_name transcoder \
--dump_path 'dump' \
--lgs 'java_sa-python_sa' \
--bt_steps 'python_sa-java_sa-python_sa,java_sa-python_sa-java_sa' \
--ae_steps 'python_sa,java_sa' \
--mt_steps 'java_sa-python_sa,python_sa-java_sa' \
--encoder_only False \
--emb_dim 1024 \
--n_heads 8 \
--n_layers 0 \
--n_layers_encoder 6 \
--n_layers_decoder 6 \
--eval_bleu true \
--eval_computation true \
--has_sentences_ids true
```
Thank you.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.