facebookresearch / facebookresearch/CodeGen

Possible overfitting with "TransCoder_model_1.pth" from Java -> Python

Open
#103 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
777
Forks
144
PR merge metrics
No merged PRs in 30d

Description

I have been using TransCoder for a few research papers lately and I came across an interesting phenomenon using the pretrained model "TransCoder_model_1.pth" you provide in your docs.

For the program in your dataset `COUNT_OF_PAIRS_SATISFYING_THE_GIVEN_CONDITION.java` when translated to Python, it passes all test cases. We checked and the translation seems correct. However, when we perform any small syntactical change to the original Java code and try to translate again, the translation mismatches the changed program.

For example, we introduced a change to the following Java code by modifying `s.charAt(i)` -> `s.charAt(++i)` at line 5:

```java
static int f_gold( int a, int b ) {
java.lang.String s = String.valueOf( b );
int i;
for (i = 0; i < s.length(); i++) {
if (s.charAt( ++i ) != '9') {
break;
}
}
int result;
if (i == s.length()) {
result = a * s.length();
} else {
result = a * (s.length() - 1);
}
return result;
}
```

The translated code is as follows:
```python
def f_gold(cls, a, b):
""" generated source for method f_gold """
s = String.valueOf(b)
i = int()
while i < len(s):
i += 1
if s.charAt(i) != '9':
break
i += 1
result = int()
if i == len(s):
result = a * len(s)
else:
result = a * (1 - len(s))
return result
```

Notice that the changed code is rightfully translated, however, `result = a * (s.length() - 1);` is now wrongfully translated to `result = a * (1 - len(s))`.

We have generated 89 variations of this Java function using [MuJava](https://cs.gmu.edu/~offutt/mujava/), and this translation bug appears in every single one of them, except the original program.

Maybe this is an overfitting example. What are your thoughts?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.