Caterina leaves UDIEN and UDINT non-zero

I have some code that does not use USB but does have some interrupt handlers that were refusing to fire. Turns out the code works fine if I include USB stuff but not if I don't. After much digging, it seems that Caterina leaves UDIEN=0x09 and UDINT=0x14 after exiting. I think it should leave USB in the state it would be in immediately after a reset, in particular with these two registers containing 0.