Make fileinfo recognize UTF-8, UTF-16, and UTF-32 strings and emit output in UTF-8
- Dominant language
- C++
- Stars
- 8.6k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
At present, `retdec-fileinfo` emits output encoded in ASCII. When it encounters characters that are not ASCII, it escapes them (e.g. `\xd0\x9e` instead of UTF-16-encoded `О`). We should consider making `retdec-fileinfo` recognize UTF-8, UTF-16, and UTF-32 strings and emit output in UTF-8 (i.e. transcode such encodings into UTF-8 so it can emit output in a single, pre-defined encoding).
I have selected these encodings because they are the most common Unicode encodings. We could also consider other, non-Unicode encodings, such as ISO-8862 or CP1250, but there are many of them. So, I believe it would be easier to start with UTF-8, UTF-16, and UTF-32.
Use cases:
1. File `0000701d1ee3f6a21d51f822aef277f0854a783a3b1d2affc5b832f3aaafab40` has the following UTF-8-encoded signer: `ООО «Аппсет»`. `retdec-fileinfo` currently emits that as `\xd0\x9e\xd0\x9e\xd0\x9e \xc2\xab\xd0\x90\xd0\xbf\xd0\xbf\xd1\x81\xd0\xb5\xd1\x82\xc2\xbb`.
2. File `009e7d577abe94162e78ba281ffd9f3e33b7ecc82b95cbf171baf6a6c2e0140f` has the following UTF-8-encoded PDB path: `E:\CPlusProject\trunk\迅捷便签安装包\Release\SetupNew.pdb`. `retdec-fileinfo` currently emits that as `E:\CPlusProject\trunk\\xe8\xbf\x85\xe6\x8d\xb7\xe4\xbe\xbf\xe7\xad\xbe\xe5\xae\x89\xe8\xa3\x85\xe5\x8c\x85\Release\SetupNew.pdb`.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running retdec-fileinfo on the two sample files named in the issue and trace the retdec-fileinfo output path responsible for escaping non-ASCII characters. Determine how UTF-8, UTF-16, and UTF-32 strings are identified and transcoded, then verify that the signer and PDB path are emitted as UTF-8 without byte escapes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- cli
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100