Feature Request/Idea: store user-specified encoding in metadata

Overview of the Feature Request

Different systems can have different default encodings, which hinders reproducibility across these machines. Storing the encoding of a file as metadata helps alleviate this. Depositors should have a UI and API option for specifying the encoding of their files. This should appear as metadata to users who want to download the file.

What kind of user is the feature intended for?

Depositor and downloader

What inspired the request?

This file cannot be decoded.

>>> f = open("Catalinac_3_Running_Wordfish.R"); f.read()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x81 in position 3830: invalid start byte

Even chardet gets confused:

>>> chardet.detect(open("Catalinac_3_Running_Wordfish.R", "rb").read())
{'encoding': 'Windows-1252', 'confidence': 0.73, 'language': ''}
>>> f = open("/home/sam/Downloads/Catalinac_3_Running_Wordfish.R", encoding="Windows-1252"); f.read()
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 3830: character maps to <undefined>

This is not an invalid file; Emacs correctly identifies the encoding as Shift JIS

>>> f = open("Catalinac_3_Running_Wordfish.R", encoding="shift-jis"); len(f.read())
3862

While R can run this file, since the 0x81 appears after a comment symbol, one can easily imagine the comment symbol being a part of the source string. This is a problem with text files in general, not R specifically. While one could use chardet, chardet can be wrong, as we see above, and it is more elegant to store the encoding separately.

The issue names no files or tests. Start by tracing the depositor UI and API paths for file metadata, then examine the metadata exposed to downloaders. Done means depositors can specify an encoding through the UI and API, and users can see that encoding as file metadata.