avast / avast/retdec

Make fileinfo recognize UTF-8, UTF-16, and UTF-32 strings and emit output in UTF-8

Open
#640 1 comment 0 reactions 0 assignees View on GitHub
C-fileinfo new-feature P-output
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

At present, `retdec-fileinfo` emits output encoded in ASCII. When it encounters characters that are not ASCII, it escapes them (e.g. `\xd0\x9e` instead of UTF-16-encoded `О`). We should consider making `retdec-fileinfo` recognize UTF-8, UTF-16, and UTF-32 strings and emit output in UTF-8 (i.e. transcode such encodings into UTF-8 so it can emit output in a single, pre-defined encoding).

I have selected these encodings because they are the most common Unicode encodings. We could also consider other, non-Unicode encodings, such as ISO-8862 or CP1250, but there are many of them. So, I believe it would be easier to start with UTF-8, UTF-16, and UTF-32.

Use cases:

1. File `0000701d1ee3f6a21d51f822aef277f0854a783a3b1d2affc5b832f3aaafab40` has the following UTF-8-encoded signer: `ООО «Аппсет»`. `retdec-fileinfo` currently emits that as `\xd0\x9e\xd0\x9e\xd0\x9e \xc2\xab\xd0\x90\xd0\xbf\xd0\xbf\xd1\x81\xd0\xb5\xd1\x82\xc2\xbb`.
2. File `009e7d577abe94162e78ba281ffd9f3e33b7ecc82b95cbf171baf6a6c2e0140f` has the following UTF-8-encoded PDB path: `E:\CPlusProject\trunk\迅捷便签安装包\Release\SetupNew.pdb`. `retdec-fileinfo` currently emits that as `E:\CPlusProject\trunk\\xe8\xbf\x85\xe6\x8d\xb7\xe4\xbe\xbf\xe7\xad\xbe\xe5\xae\x89\xe8\xa3\x85\xe5\x8c\x85\Release\SetupNew.pdb`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running retdec-fileinfo on the two sample files named in the issue and trace the retdec-fileinfo output path responsible for escaping non-ASCII characters. Determine how UTF-8, UTF-16, and UTF-32 strings are identified and transcoded, then verify that the signer and PDB path are emitted as UTF-8 without byte escapes.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.