PBXFileReference not reporting all referrers when file is a .xib

When trying to remove files from a project I've noticed that objects in this format:
``
do not report all of their referrers.

Calling referrers on that object returns one object, which I remove, but then when trying to remove that original object it says there is still a reference to a PBXBuildFile. For .m and .h files all referrers are being reported properly, but it does not seem to be working correctly with .xibs

Is there a different method for removing .xibs from a project than #remove_from_project?