corelight / corelight/cwrap

Will `cwrap` work with `Nim`?

Open
#2 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Perl
Stars
34
Forks
2
PR merge metrics
No merged PRs in 30d

Description

Yes, here's an example:

* Clone and build Nim:

```
$ git clone https://github.com/nim-lang/Nim.git
$ pushd ./Nim/
$ time ./build_all.sh
...
real 4m24.324s
$ popd

$ ./Nim/bin/nim --version | egrep -i nim
Nim Compiler Version 1.7.1 [Linux: amd64]
```

* Create an example Nim source file, and build and run it without `cwrap`:

```
$ cat helloworld.nim
proc foo(): void =
echo "Hello World from foo()"

proc bar(): void =
foo()

echo "Hello World 2"
bar()

$ ./Nim/bin/nim c -r --verbosity:2 helloworld.nim
...
CC: helloworld.nim: gcc -c -w -fmax-errors=3 -pthread -I/home/simon/work/20220722-nim/Nim/lib -I/home/simon/work/20220722-nim -o /home/simon/.cache/nim/helloworld_d/@mhelloworld.nim.c.o /home/simon/.cache/nim/helloworld_d/@mhelloworld.nim.c
Hint: gcc -o /home/simon/work/20220722-nim/helloworld /home/simon/.cache/nim/helloworld_d/@mNim@slib@sstd@sprivate@sdigitsutils.nim.c.o /home/simon/.cache/nim/helloworld_d/@mNim@slib@ssystem@sdollars.nim.c.o /home/simon/.cache/nim/helloworld_d/@mNim@slib@spure@scollections@ssharedlist.nim.c.o /home/simon/.cache/nim/helloworld_d/@mNim@slib@sstd@ssyncio.nim.c.o /home/simon/.cache/nim/helloworld_d/@mNim@slib@ssystem.nim.c.o /home/simon/.cache/nim/helloworld_d/@mhelloworld.nim.c.o -pthread -ldl [Link]
/home/simon/work/20220722-nim/Nim/compiler/extccomp.nim(402, 13) compiler msg initiated here [MsgOrigin]
...
Hello World 2
Hello World from foo()

$ wc -c helloworld
116680 helloworld
```

* Try to tell Nim to use a different compiler:

```
$ ./Nim/bin/nim c --cc:xyz -r --verbosity:2 helloworld.nim
...
command line(1, 2) Error: unknown C compiler: 'xyz'. Available options are: gcc, switch_gcc, llvm_gcc, clang, bcc, vcc, tcc, env, icl, icc, clang_cl
```

* To get `cwrap` to wrap `Nim` compilation, we're going to have to trick Nim into thinking we want to to use one of its supported compilers, e.g. `icc`:

```
$ which icc
$

$ ./Nim/bin/nim c --cc:icc -r --verbosity:2 helloworld.nim
...
Error: execution of an external compiler program 'icc -c -pthread -I/home/simon/work/20220722-nim/Nim/lib -I/home/simon/work/20220722-nim -o /home/simon/.cache/nim/helloworld_d/@mNim@slib@sstd@sprivate@sdigitsutils.nim.c.o /home/simon/.cache/nim/helloworld_d/@mNim@slib@sstd@sprivate@sdigitsutils.nim.c' failed with exit code: 127
```

* Create a dummy `icc` "compiler" for Nim to run:

```
$ PATH=$PATH:`pwd`

$ cat icc
echo "hello from icc"
exit 1

$ chmod +x icc

$ ./Nim/bin/nim c --cc:icc --forceBuild:on -r --verbosity:2 helloworld.nim
...
Error: execution of an external compiler program 'icc -c -pthread -I/home/simon/work/20220722-nim/Nim/lib -I/home/simon/work/20220722-nim -o /home/simon/.cache/nim/helloworld_d/@mNim@slib@sstd@sprivate@sdigitsutils.nim.c.o /home/simon/.cache/nim/helloworld_d/@mNim@slib@sstd@sprivate@sdigitsutils.nim.c' failed with exit code: 1
...
hello from icc
```

* Clone `cwrap` and install its dependencies:

```
$ git clone $ https://github.com/corelight/cwrap.git

$ sudo apt-get install liblist-moreutils-perl

$ sudo apt install cpanminus

$ sudo cpanm FFI::Platypus::Lang::CPP::Demangle::XS

$ sudo apt install llvm
$ which llvm-cxxfilt
/usr/bin/llvm-cxxfilt
```

* Modify `icc` fake compiler to point to `cwrap` (which uses `gcc` under the covers):

```
$ cat icc
`pwd`/cwrap/cwrap.pl "$@"
```

* Hot patch a small bug in `cwrap`:

```
$ git diff
diff --git a/cwrap.pl b/cwrap.pl
- $gcc_4_s =~ s~$source_file_argument~$s_file.2.s~;
+ if ($gcc_4_s =~ s~$source_file_argument[ ]~$s_file.2.s ~) {} # if source file is not at the end of the command line
+ elsif ($gcc_4_s =~ s~$source_file_argument$~$s_file.2.s~) {} # if source file is at the end of the command line
+ else { cwrap_die(sprintf qq[%f ERROR: cwrap regex failed to substitute source files; gcc_4_s=%s!\n], Time::HiRes::time() - $ts, $gcc_4_s); }
```

* Tell `Nim` to force re-build `helloworld.nim` using `icc`:

```
$ ./Nim/bin/nim c --cc:icc --forceBuild:on -r --verbosity:2 helloworld.nim
...
Hello World 2
Hello World from foo()

$ wc -c helloworld
226696 helloworld
```

* `helloworld` executable is now bigger because 100s of functions are instrumented by `cwrap`.
* Run `helloworld` and ask `cwrap` to list all the instruction function (note: many functions are grepped away for brevity):

```
$ CWRAP_LOG_SHOW=1 ./helloworld | egrep -v "(system|sdigitsutils|ssyncio|ssharedlist)"
C0 + cwrap_log_show() { #1
C0 - func_addr=(nil)
C0 - #1: verbosity 9 for 1 of 1 function variation(s) for cwrap_log_show() from helloworld.cwrap.c
C0 - #2: verbosity 9 for 1 of 1 function variation(s) for cwrap_log_stats() from helloworld.cwrap.c
C0 - #3: verbosity 9 for 1 of 1 function variation(s) for cwrap_log_verbosity_set() from helloworld.cwrap.c
C0 - #4: verbosity 9 for 1 of 1 function variation(s) for cwrap_log_quiet_until() from helloworld.cwrap.c
C0 - #107: verbosity 9 for 1 of 1 function variation(s) for main() from /home/simon/.cache/nim/helloworld_d/@mhelloworld.nim.c.o
C0 - #126: verbosity 9 for 1 of 1 function variation(s) for initStackBottomWith() from /home/simon/.cache/nim/helloworld_d/@mhelloworld.nim.c.o
C0 - #153: verbosity 9 for 1 of 1 function variation(s) for foo__helloworld_1() from /home/simon/.cache/nim/helloworld_d/@mhelloworld.nim.c.o
C0 - #194: verbosity 9 for 1 of 1 function variation(s) for bar__helloworld_2() from /home/simon/.cache/nim/helloworld_d/@mhelloworld.nim.c.o
C0 - #225: verbosity 9 for 1 of 1 function variation(s) for PreMainInner() from /home/simon/.cache/nim/helloworld_d/@mhelloworld.nim.c.o
C0 - #226: verbosity 9 for 1 of 1 function variation(s) for PreMain() from /home/simon/.cache/nim/helloworld_d/@mhelloworld.nim.c.o
C0 - #227: verbosity 9 for 1 of 1 function variation(s) for NimMainModule() from /home/simon/.cache/nim/helloworld_d/@mhelloworld.nim.c.o
C0 - #228: verbosity 9 for 1 of 1 function variation(s) for NimMainInner() from /home/simon/.cache/nim/helloworld_d/@mhelloworld.nim.c.o
C0 - #229: verbosity 9 for 1 of 1 function variation(s) for NimMain() from /home/simon/.cache/nim/helloworld_d/@mhelloworld.nim.c.o
C0 } // cwrap_log_show()
```

* Run `helloworld` and ask `cwrap` to run with full verbosity except for functions in `ssystem.nim.c` and `sdigitsutils.nim.c`:

```
$ CWRAP_LOG_STATS=1 CWRAP_LOG_NUM=1 CWRAP_LOG_TIMESTAMP=1 CWRAP_LOG_THREAD_ID=1 CWRAP_LOG_CURT=1 CWRAP_LOG_VERBOSITY_SET=1/9=file-ssystem.nim.c/9=file-sdigitsutils.nim.c ./helloworld | less +G
cwrap_log_init() {} // CWRAP_LOG: _VERBOSITY_SET=1/9=file-ssystem.nim.c/9=file-sdigitsutils.nim.c ([={file|function}-][/...]) _QUIET_UNTIL=(null) _STATS=1 _SHOW=0 _CURT=1 _FILE=0 _NUM=1 _COR_ID=1 _LIMIT=10,000 _THREAD_ID=1 _STACK_PTR=0
#1 T06822 C0 0.000011s + cwrap_log_verbosity_set(verbosity=1/9=file-ssystem.nim.c/9=file-sdigitsutils.nim.c) { // #1 [cwrap_log_verbosity_set() ignores verbosity!]
#2 T06822 C0 0.000029s - verbosity 1 set for 240 matches in 240 functions for 1 byte clause '1' // type=FILE|FUNCTION keyword=(null)
#3 T06822 C0 0.000031s - verbosity 9 set for 210 matches in 240 functions for 20 byte clause '9=file-ssystem.nim.c' // type=FILE keyword=ssystem.nim.c
#4 T06822 C0 0.000032s - verbosity 9 set for 10 matches in 240 functions for 25 byte clause '9=file-sdigitsutils.nim.c' // type=FILE keyword=sdigitsutils.nim.c
#5 T06822 C0 0.000034s } // cwrap_log_verbosity_set()
#6 T06822 C0 0.000036s + main() { // #1
#7 T06822 C0 0.000039s + NimMain() { // #1
#8 T06822 C0 0.000040s + PreMain() { // #1
#9 T06822 C0 0.000042s + initStackBottomWith() {} // #1
#10 T06822 C0 0.000102s + init__system_6081() { // #1
#11 T06822 C0 0.000105s + initLock__coreZlocks_64() {} // #1
#12 T06822 C0 0.000106s } // init__system_6081()
#13 T06822 C0 0.000109s + PreMainInner() {} // #1
#14 T06822 C0 0.000110s } // PreMain()
#15 T06822 C0 0.000111s + initStackBottomWith() {} // #2
#16 T06822 C0 0.000112s + NimMainInner() { // #1
#17 T06822 C0 0.000113s + NimMainModule() { // #1
#18 T06822 C0 0.000114s + echoBinSafe() { // #1
#19 T06822 C0 0.000115s + nimToCStringConv() {} // #1
Hello World 2
#20 T06822 C0 0.000120s } // echoBinSafe()
#21 T06822 C0 0.000121s + bar__helloworld_2() { // #1
#22 T06822 C0 0.000122s + foo__helloworld_1() { // #1
#23 T06822 C0 0.000122s + echoBinSafe() { // #2
#24 T06822 C0 0.000123s + nimToCStringConv() {} // #2
Hello World from foo()
#25 T06822 C0 0.000125s } // echoBinSafe()
#26 T06822 C0 0.000126s } // foo__helloworld_1()
#27 T06822 C0 0.000126s } // bar__helloworld_2()
#28 T06822 C0 0.000127s } // NimMainModule()
#29 T06822 C0 0.000128s } // NimMainInner()
#30 T06822 C0 0.000129s } // NimMain()
#31 T06822 C0 0.000129s } // main()
#32 T06822 C0 0.000131s + cwrap_log_stats() { // #1 [cwrap_log_stats() ignores verbosity!]
#33 T06822 C0 0.000132s - 1 calls to 1 of 1 function variation(s) for cwrap_log_stats()
#34 T06822 C0 0.000133s - 1 calls to 1 of 1 function variation(s) for cwrap_log_verbosity_set()
#35 T06822 C0 0.000135s - 2 calls to 1 of 1 function variation(s) for nimToCStringConv()
#36 T06822 C0 0.000136s - 1 calls to 1 of 1 function variation(s) for main()
#37 T06822 C0 0.000137s - 1 calls to 1 of 1 function variation(s) for init__system_6081()
#38 T06822 C0 0.000138s - 2 calls to 1 of 1 function variation(s) for initStackBottomWith()
#39 T06822 C0 0.000139s - 1 calls to 1 of 1 function variation(s) for initLock__coreZlocks_64()
#40 T06822 C0 0.000140s - 1 calls to 1 of 1 function variation(s) for foo__helloworld_1()
#41 T06822 C0 0.000140s - 2 calls to 1 of 1 function variation(s) for echoBinSafe()
#42 T06822 C0 0.000142s - 1 calls to 1 of 1 function variation(s) for bar__helloworld_2()
#43 T06822 C0 0.000143s - 1 calls to 1 of 1 function variation(s) for PreMainInner()
#44 T06822 C0 0.000143s - 1 calls to 1 of 1 function variation(s) for PreMain()
#45 T06822 C0 0.000144s - 1 calls to 1 of 1 function variation(s) for NimMainModule()
#46 T06822 C0 0.000145s - 1 calls to 1 of 1 function variation(s) for NimMainInner()
#47 T06822 C0 0.000146s - 1 calls to 1 of 1 function variation(s) for NimMain()
#48 T06822 C0 0.000147s - 18 calls to 15 of 240 functions instrumented
#49 T06822 C0 0.000148s } // cwrap_log_stats()
```

* We get to see a hint into the inner life of the `helloworld` Nim executable.
* Run `helloworld` and ask `cwrap` to run with no verbosity until function `NimMainInner()` is called for the first time.
* Note: Now we can see a but more detail, e.g. previously hidden functions, e.g. `nimFrame()`:

```
$ CWRAP_LOG_QUIET_UNTIL=NimMainInner CWRAP_LOG_STATS=1 CWRAP_LOG_NUM=1 CWRAP_LOG_TIMESTAMP=1 CWRAP_LOG_THREAD_ID=1 CWRAP_LOG_CURT=1 CWRAP_LOG_VERBOSITY_SET=1 ./helloworld
cwrap_log_init() {} // CWRAP_LOG: _VERBOSITY_SET=1 ([={file|function}-][/...]) _QUIET_UNTIL=NimMainInner _STATS=1 _SHOW=0 _CURT=1 _FILE=0 _NUM=1 _COR_ID=1 _LIMIT=10,000 _THREAD_ID=1 _STACK_PTR=0 _TIMESTAMP=1 _UNWIND=0 _ON_VALGRIND=0
#1 T06837 C0 0.000036s + cwrap_log_verbosity_set(verbosity=1) { // #1 [cwrap_log_verbosity_set() ignores verbosity!]
#2 T06837 C0 0.000055s - verbosity 1 set for 240 matches in 240 functions for 1 byte clause '1' // type=FILE|FUNCTION keyword=(null)
#3 T06837 C0 0.000062s } // cwrap_log_verbosity_set()
#4 T06837 C0 0.000064s + cwrap_log_quiet_until(name=NimMainInner) {} // #1 going quiet until function NimMainInner() [cwrap_log_quiet_until() ignores verbosity!]
#5 T06837 C0 0.000127s + NimMainInner() { // #1
#6 T06837 C0 0.000129s + NimMainModule() { // #1
#7 T06837 C0 0.000147s + nimFrame() {} // #1
#8 T06837 C0 0.000150s + echoBinSafe() { // #1
#9 T06837 C0 0.000151s + nimFrame() {} // #2
#10 T06837 C0 0.000154s + nimToCStringConv() {} // #1
Hello World 2
#11 T06837 C0 0.000156s + nimAddInt() {} // #1
#12 T06837 C0 0.000158s + popFrame() {} // #1
#13 T06837 C0 0.000159s } // echoBinSafe()
#14 T06837 C0 0.000161s + bar__helloworld_2() { // #1
#15 T06837 C0 0.000162s + nimFrame() {} // #3
#16 T06837 C0 0.000164s + foo__helloworld_1() { // #1
#17 T06837 C0 0.000165s + nimFrame() {} // #4
#18 T06837 C0 0.000167s + echoBinSafe() { // #2
#19 T06837 C0 0.000168s + nimFrame() {} // #5
#20 T06837 C0 0.000170s + nimToCStringConv() {} // #2
Hello World from foo()
#21 T06837 C0 0.000172s + nimAddInt() {} // #2
#22 T06837 C0 0.000174s + popFrame() {} // #2
#23 T06837 C0 0.000175s } // echoBinSafe()
#24 T06837 C0 0.000177s + popFrame() {} // #3
#25 T06837 C0 0.000178s } // foo__helloworld_1()
#26 T06837 C0 0.000180s + popFrame() {} // #4
#27 T06837 C0 0.000181s } // bar__helloworld_2()
#28 T06837 C0 0.000183s + popFrame() {} // #5
#29 T06837 C0 0.000185s } // NimMainModule()
#30 T06837 C0 0.000187s } // NimMainInner()
#31 T06837 C0 0.000189s } // NimMain()
#32 T06837 C0 0.000206s } // main()
#33 T06837 C0 0.000208s + colonanonymous___system_5714() {} // #1
#34 T06837 C0 0.000221s + cwrap_log_stats() { // #1 [cwrap_log_stats() ignores verbosity!]
#35 T06837 C0 0.000223s - 1 calls to 1 of 1 function variation(s) for cwrap_log_stats()
#36 T06837 C0 0.000226s - 1 calls to 1 of 1 function variation(s) for cwrap_log_verbosity_set()
#37 T06837 C0 0.000228s - 1 calls to 1 of 1 function variation(s) for cwrap_log_quiet_until()
#38 T06837 C0 0.000230s - 5 calls to 1 of 1 function variation(s) for popFrame()
#39 T06837 C0 0.000233s - 2 calls to 1 of 1 function variation(s) for nimToCStringConv()
#40 T06837 C0 0.000234s - 5 calls to 1 of 1 function variation(s) for nimFrame()
#41 T06837 C0 0.000236s - 2 calls to 1 of 1 function variation(s) for nimAddInt()
#42 T06837 C0 0.000239s - 1 calls to 1 of 1 function variation(s) for foo__helloworld_1()
#43 T06837 C0 0.000242s - 2 calls to 1 of 1 function variation(s) for echoBinSafe()
#44 T06837 C0 0.000244s - 1 calls to 1 of 1 function variation(s) for colonanonymous___system_5714()
#45 T06837 C0 0.000246s - 1 calls to 1 of 1 function variation(s) for bar__helloworld_2()
#46 T06837 C0 0.000248s - 1 calls to 1 of 1 function variation(s) for NimMainModule()
#47 T06837 C0 0.000249s - 1 calls to 1 of 1 function variation(s) for NimMainInner()
#48 T06837 C0 0.000251s - 24 calls to 13 of 240 functions instrumented
#49 T06837 C0 0.000253s } // cwrap_log_stats()
```

* That's it :-)

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the Nim example using the fake `icc` compiler and the `cwrap.pl` entry point. Inspect the source-file substitution behavior shown by the hot patch, then verify that a Nim-generated executable builds, runs, and reports instrumented functions without requiring a local patch.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, perl
Domain
devtools
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.