avast / avast/retdec

Wrong decompilation of "rep stos"

Open
#832 2 comments 0 reactions 0 assignees View on GitHub
bug C-capstone2llvmir
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

Subject: https://github.com/rfalke/decompiler-subjects/blob/master/from_holdec/i386_memset_rep_stos/ia32_elf/subject.exe

Issues:
* `rep stosw` generates the same decompiler output as `rep stowd`. See functions `i386_memset_forward_w()` and `i386_memset_forward_l()`. Maybe one of the `__asm_rep_stosd_memset()` should be `__asm_rep_stosW_memset()`?
* retdec doesn't take into account the value of the direction flag. While it is a convention that it is cleared it doesn't have to be. The functions with `backward` or `both` in name show this.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the referenced subject.exe and compare the decompiler output for rep stosw and rep stosd in i386_memset_forward_w() and i386_memset_forward_l(). Then inspect the backward and both functions for direction-flag handling. Done means word and double-word stores decompile distinctly and the direction flag is accounted for rather than assumed clear.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers, reverse-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.