eclipse-jdt / eclipse-jdt/eclipse.jdt.core
Eclipse does not properly normalize identifiers according to the JLS
- Dominant language
- Java
- Stars
- 237
- Forks
- 195
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 49
Description
Example:
Note this is the visual representation from my vim editor so I've attached the source to preserve the actual bytes.
```
public class Test
{
public static void test() {
int variable;
variable^Z = 5;
System.err.println(var<200b>^Ziable^Z^Z);
}
}
```
(I had to throw this into a zip because the raw Test.java file was rejected by github's file uploader)
[Test.java.zip](https://github.com/eclipse-jdt/eclipse.jdt.core/files/13079245/Test.java.zip)
(<200b> corresponds to u+200b or a zero width space, ^Z corresponds to u+001a, the sub char - just used as an example of a different control character that is ignorable in identifiers).
Javac handles the attached file just fine, but Eclipse fails to match the varying representations for 'variable'.
Aside from javac, since I understand that matching javac may be less important than the JLS (though I don't know what guarantees Eclipse makes wrt matching the spec), the JLS is explicit about this:
https://docs.oracle.com/javase/specs/jls/se21/html/jls-3.html#jls-IdentifierChars
> Two identifiers are the same only if, after ignoring characters that are ignorable, the identifiers have the same Unicode character for each letter or digit. An ignorable character is a character for which the method Character.isIdentifierIgnorable(int) returns true. Identifiers that have the same external appearance may yet be different.
So Eclipse does not fail to parse these identifiers as a result of using Character.isJavaIdentifierPart during the scanning phase, but it does fail to normalize when bytes differ only wrt ignorable characters. Accordingly, they are not treated as one and the same which causes Eclipse to fail. Note that the example is obviously contrived but I have seen corporate customers use u+200b (zero-width spaces) to enable some external spell checker on their identifiers and it would be pretty easy to differ and not know it.
Contributor guide
Assessment
This issue has not been assessed yet.