Needs smoke tests

#480 and #481 fixed an issue that I introduced by referring to a constant that wasn't loaded yet.

CI didn't fail, even though every app failed the moment it tried to load.

I think the solution here is to add a smoke test that runs an app of some kind (Rails/Sinatra/pure Rack) and make sure the app starts. This actually might be more effective than all of the specs we have, which currently have to cover a lot of different cases (with Rails, each version of rails, etc).