Testing vim-plugin with compiler script results in "Vim(compiler):E666: compiler not supported; xxx"

I'm working on a plugin to work with [meson](https://mesonbuild.com) and one of the features I currently implementing is support for `CompilerSet errorformat`. In my case [ninja](https://ninja-build.org/).
To enable the compiler I have in my plugin/*.vim file
```
let &makeprg = 'ninja -C builddir'
silent execute ':compiler! ninja`
```
Now you can type `:make` and the program is compiled.
But when I run my tests I get an error `Vim(compiler):E666: compiler not supported: ninja`.

Here is my vimrc:
```
filetype off

set rtp+=vader.vim
set rtp+=.
set rtp+=after

filetype plugin indent on
syntax enable
```
Do I need to set up more or different in my vimrc?
Is it even possible to test scripts with compiler support?

Here is a complete log of running one of my tests run with comman `vim -Nu ./test/.vimrc -c 'Vader! test/meson-compile-project.vader':
```
The Meson build system
Version: 0.54.0
Source dir: /home/jenne/devel/GH/j-carl/vim-mesonist/test/meson-project
Build dir: /home/jenne/devel/GH/j-carl/vim-mesonist/test/meson-project/builddir
Build type: native build
Project name: test-compile
Project version: undefined
C++ compiler for the host machine: ccache c++ (gcc 9.3.0 "c++ (Arch Linux 9.3.0-1) 9.3.0")
C++ linker for the host machine: c++ ld.bfd 2.34
Host machine cpu family: x86_64
Host machine cpu: x86_64
Build targets in project: 1

Found ninja-1.10.0 at /usr/bin/ninja
The Meson build system
Version: 0.54.0
Source dir: /home/jenne/devel/GH/j-carl/vim-mesonist/test/meson-project
Build dir: /home/jenne/devel/GH/j-carl/vim-mesonist/test/meson-project/builddir
Build type: native build
Project name: test-compile
Project version: undefined
C++ compiler for the host machine: ccache c++ (gcc 9.3.0 "c++ (Arch Linux 9.3.0-1) 9.3.0")
C++ linker for the host machine: c++ ld.bfd 2.34
Host machine cpu family: x86_64
Host machine cpu: x86_64
Build targets in project: 1

Found ninja-1.10.0 at /usr/bin/ninja
Vader note: cannot print to stderr reliably/directly. Please consider using Vim's -es/-Es option (mode=n).
VIM - Vi IMproved 8.2 (2019 Dec 12, compiled Apr 4 2020 13:23:00)
Included patches: 1-510
Compiled by Arch Linux
Huge version without GUI. Features included (+) or not **(-):
+acl +cmdline_info -farsi +libcall -mouse_sysmouse +python/dyn -tag_old_static +visual
+arabic +comments +file_in_path +linebreak +mouse_urxvt +python3/dyn -tag_any_white +visualextra
+autocmd +conceal +find_in_path +lispindent +mouse_xterm +quickfix +tcl/dyn +viminfo
+autochdir +cryptv +float +listcmds +multi_byte +reltime +termguicolors +vreplace
-autoservername +cscope +folding +localmap +multi_lang +rightleft +terminal +wildignore
-balloon_eval +cursorbind -footer +lua/dyn -mzscheme +ruby/dyn +terminfo +wildmenu
+balloon_eval_term +cursorshape +fork() +menu +netbeans_intg +scrollbind +termresponse +windows
-browse +dialog_con +gettext +mksession +num64 +signs +textobjects +writebackup
++builtin_terms +diff -hangul_input +modify_fname +packages +smartindent +textprop -X11
+byte_offset +digraphs +iconv +mouse +path_extra -sound +timers -xfontset
+channel -dnd +insert_expand -mouseshape +perl/dyn +spell +title -xim
+cindent -ebcdic +job +mouse_dec +persistent_undo +startuptime -toolbar -xpm
-clientserver +emacs_tags +jumplist +mouse_gpm +popupwin +statusline +user_commands -xsmp
-clipboard +eval +keymap -mouse_jsbterm +postscript -sun_workshop +vartabs -xterm_clipboard
+cmdline_compl +ex_extra +lambda +mouse_netterm +printer +syntax +vertsplit -xterm_save
+cmdline_hist +extra_search +langmap +mouse_sgr +profile +tag_binary +virtualedit
system vimrc file: "/etc/vimrc"
user vimrc file: "$HOME/.vimrc"
2nd user vimrc file: "~/.vim/vimrc"
user exrc file: "$HOME/.exrc"
defaults file: "$VIMRUNTIME/defaults.vim"
fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: gcc -L. -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,-E -Wl,-rpath,/usr/lib/perl5/5.
30/core_perl/CORE -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -L/usr/local/lib -Wl,--as-needed -o vim -lm -ltinfo -lelf -lnsl -lacl -lattr -lg
pm -ldl -Wl,-E -Wl,-rpath,/usr/lib/perl5/5.30/core_perl/CORE -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -fstack-protector-strong -L/usr/local/lib -L/
usr/lib/perl5/5.30/core_perl/CORE -lperl -lpthread -ldl -lm -lcrypt -lutil -lc -L/usr/lib -ltclstub8.6 -ldl -lz -lpthread -lm
Starting Vader: 1 suite(s), 4 case(s)
Starting Vader: /home/jenne/devel/GH/j-carl/vim-mesonist/test/meson-compile-project.vader
(1/4) [EXECUTE] Check commands
(2/4) [EXECUTE] Default values
(3/4) [EXECUTE] Setup meson project
(3/4) [EXECUTE] (X) Vim(compiler):E666: compiler not supported: ninja
> function 30_MesonistSetup, line 32
(4/4) [EXECUTE] Compile meson project
(4/4) [EXECUTE] (X) Vim(compiler):E666: compiler not supported: ninja
> function 30_MesonistSetup, line 32
(4/4) [ EXPECT] (X)
- Expected:

Hello Meson
- Got:
project('test-compile', 'cpp')

executable('test-compile', 'test_compile.cpp')
Success/Total: 2/4
Success/Total: 2/4 (assertions: 3/3)
Elapsed time: 0.97 sec.
```