facebookresearch / facebookresearch/CodeGen
Possible overfitting with "TransCoder_model_1.pth" from Java -> Python
- Dominant language
- Python
- Stars
- 777
- Forks
- 144
- PR merge metrics
- No merged PRs in 30d
Description
I have been using TransCoder for a few research papers lately and I came across an interesting phenomenon using the pretrained model "TransCoder_model_1.pth" you provide in your docs.
For the program in your dataset `COUNT_OF_PAIRS_SATISFYING_THE_GIVEN_CONDITION.java` when translated to Python, it passes all test cases. We checked and the translation seems correct. However, when we perform any small syntactical change to the original Java code and try to translate again, the translation mismatches the changed program.
For example, we introduced a change to the following Java code by modifying `s.charAt(i)` -> `s.charAt(++i)` at line 5:
```java
static int f_gold( int a, int b ) {
java.lang.String s = String.valueOf( b );
int i;
for (i = 0; i < s.length(); i++) {
if (s.charAt( ++i ) != '9') {
break;
}
}
int result;
if (i == s.length()) {
result = a * s.length();
} else {
result = a * (s.length() - 1);
}
return result;
}
```
The translated code is as follows:
```python
def f_gold(cls, a, b):
""" generated source for method f_gold """
s = String.valueOf(b)
i = int()
while i < len(s):
i += 1
if s.charAt(i) != '9':
break
i += 1
result = int()
if i == len(s):
result = a * len(s)
else:
result = a * (1 - len(s))
return result
```
Notice that the changed code is rightfully translated, however, `result = a * (s.length() - 1);` is now wrongfully translated to `result = a * (1 - len(s))`.
We have generated 89 variations of this Java function using [MuJava](https://cs.gmu.edu/~offutt/mujava/), and this translation bug appears in every single one of them, except the original program.
Maybe this is an overfitting example. What are your thoughts?
Contributor guide
Assessment
This issue has not been assessed yet.