eclipse-platform / eclipse-platform/eclipse.platform

content-type extension point "default-charset" is used for overwriting existing (e.g. BOM+UTF-16) files

Open
#244 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
165
Forks
174
Avg merge
2d 8h
Merged PRs (30d)
22

Description

Defining a [content-type](https://help.eclipse.org/latest/topic/org.eclipse.platform.doc.isv/reference/extension-points/org_eclipse_core_contenttype_contentTypes.html?cp=2_1_1_9#e.content-type) with a `default-charset` attribute will cause that charset to be used when _saving_ (modified) existing files, even if those files had some other character set, e.g. BOM+`UTF-16`.

This has multiple problems (the example is a file `foo.properties`, encoded with BOM and `UTF-16`, where its content type (`org.eclipse.core.runtime.properties`) has a `default-charset` attribute specifying `ISO-8859-1`):

1. Doing a no-op (e.g. deleting and re-inserting a space) modification and saving the file will change the encoding, and thus change the entire file contents. This is a problem in many ways, e.g. for version control systems.
2. Inserting a character, e.g. the euro symbol (U+20AC), which cannot be represented in the `default-charset` (`ISO-8859-1`), and trying to save the file, will bring up a dialog about "Save Problems" which will suggest to save in `UTF-8`. The (obviously?) right thing would be to save the file in its current encoding (BOM+`UTF-16`) which _can_ represent the new contents.
3. A variant of the above, if the file _already_ contains some character (such as the euro symbol) that cannot be encoding in the `default-charset`, then doing a no-op modification and attempting to save the file will give the error dialog and refuse to save.

I encountered this in my custom content type, but it is easy to reproduce using any of the common content types that has, e.g., `ISO-8859-1` as `default-charset` attribute:

1. In Eclipse, create a plain project (no natures).
2. Outside of Eclipse, create a file `foo.properties` in the project directory, and make its contents use BOM with `UTF-16`, e.g.:
```
$ echo 'abc' | iconv -f UTF-8 -t UTF-16 > foo.properties
$ file foo.properties
foo.properties: Unicode text, UTF-16, big-endian text
```
4. Back in Eclipse, refresh the project, and open the file from the Project Explorer.

Now all the issues above can easily be reproduced.

The documentation of the `default-charset` attribute is not very illuminating, but I had expected it to be used when _opening_ a file where the file contents itself did not provide a hint. When _saving_ the file, I would always expect the current encoding, however that was determined, to be preserved.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.