avast / avast/retdec

Correct and full support of syscalls

Open
#195 1 comment 0 reactions 0 assignees View on GitHub
C-bin2llvmir enhancement P-output
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

Our current implementations of syscalls reconstruction have many problems. It needs a rewrite.

Regression tests with syscalls:
```
MIPS – bugs.732-sample-1991.Test
ARM – bugs.730-sample-8a5c.Test
x86 – bugs.761.Test
```

More binaries for testing: [test.zip](https://github.com/avast-tl/retdec/files/1703782/test.zip)

Issues:
- Handle all syscalls, not only some.
- Use stack variables if we run out of registers.
- ARM & MIPS: implement fallback dummy function as in x86.
- x86: call of `102 = socketcall` depends on value in `ebp`.
- More aggressive propagation on x86 - there were more detections in `bugs.761.Test`.
- Syscall argument propagation to stub functions.
- Syscall function should be called `sys_fnc()`. Right now we call them only `fnc()` to get type info for them. The problem is, this type info is not always the same for syscalls as for ordinary functions:
```c
int clone(int (*fn)(void *), void *child_stack,
int flags, void *arg, ...
/* pid_t *ptid, struct user_desc *tls, pid_t *ctid */ );
```

```c
long sys_clone(unsigned long flags, void *child_stack,
void *ptid, void *ctid,
struct pt_regs *regs);
```
- The problem is that it looks like some functions may have many more mutations with slightly different arguments (e.g. already mentioned `clone`).
- Different parameter passing on x86 (maybe on other archs as well?):
- x86 Linux - registers and memory reference in `ebp`
- x86 BSD (other UNIXes?) - all on stack
- other UNIXes? Can we even distinguish Linux vs BSD?
- Windows syscalls ...

Contributor guide

No contributing guide indexed for this repository

Research direction

Begin with the listed regression cases: bugs.732-sample-1991.Test, bugs.730-sample-8a5c.Test, and bugs.761.Test, then use the supplied test.zip binaries for broader coverage. Map the existing syscall reconstruction entry points and compare behavior across ARM, MIPS, x86, and Windows. Done means the listed syscall handling, argument propagation, calling conventions, and syscall naming requirements are covered without regressions.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers, operating-systems, reverse-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.