rspec-queue does not fail if there are NameErrors in specs

I've just encountered the following behaviour with the rspec runner of ci-queue:

1. Modify a spec to reference an uninitialized constant so as to trigger a NameError, e.g. by referring to a non-existent class:

describe MyNonExistentClass do
 ...
end

2. Run rspec-queue
3. rspec-queue exits successfully.

I guessed wrongly that rspec-queue would fail with a non-zero exit code, but it exited with 0.

Bare rspec exits with 1 in the above situation.

I'm wondering if this is intended behaviour - perhaps its needed for retries to work?

The error message output is:

NameError: uninitialized constant MyNonExistentClass

Debugging shows that line 400 is where the 0 exit code is returned, because the NameError causes @world.non_example_failure to be true. Syntax errors appear to have the same result:

https://github.com/Shopify/ci-queue/blob/cba2cddb6de4342fc7b5dd911f53d99651e3723e/ruby/lib/rspec/queue.rb#L387-L401

Start with ruby/lib/rspec/queue.rb around lines 387-401 and reproduce the NameError case using rspec-queue, comparing its exit status with bare rspec. Trace how NameError and syntax errors affect the reported result and determine whether the completed behavior should return a non-zero status without breaking retries; done means regression coverage and correct failure status.