Invalid dylibs added as native references prevent project from building even after removed until next clear

### Steps to Reproduce

1. Create new XM project
2. Create an invalid dylib (create a text file with "this is not a dylib" and name it foo.dylib)
3. Add it as a native reference
4. Build
5. Fails as expected.
6. Remove and build again
7. Still fails

### Expected Behavior

A successful build

### Actual Behavior
```
System.Exception: File format not recognized: /Users/donblas/Projects/TestInvalidNativeRefs/bin/Debug/TestInvalidNativeRefs.app/Contents/MonoBundle/foo.dylib (magic: 0x73696854)
at Xamarin.MachO.ReadFile (System.IO.BinaryReader reader, System.String filename) [0x00085] in /Users/builder/data/lanes/5944/64fece5f/source/xamarin-macios/tools/common/MachO.cs:183
at Xamarin.MachO.ReadFile (System.String filename) [0x00014] in /Users/builder/data/lanes/5944/64fece5f/source/xamarin-macios/tools/common/MachO.cs:191
at Xamarin.MachO+d__15.MoveNext () [0x00036] in /Users/builder/data/lanes/5944/64fece5f/source/xamarin-macios/tools/common/MachO.cs:198
at Xamarin.MachO.GetNativeDependencies (System.String libraryName) [0x000bb] in /Users/builder/data/lanes/5944/64fece5f/source/xamarin-macios/tools/common/MachO.cs:313
at Xamarin.Bundler.Driver.CopyDependencies (System.Collections.Generic.IDictionary`2[TKey,TValue] libraries) [0x001e1] in /Users/builder/data/lanes/5944/64fece5f/source/xamarin-macios/tools/mmp/driver.cs:1557
at Xamarin.Bundler.Driver.Pack (System.Collections.Generic.IList`1[T] unprocessed) [0x00581] in /Users/builder/data/lanes/5944/64fece5f/source/xamarin-macios/tools/mmp/driver.cs:842
at Xamarin.Bundler.Driver.Main2 (System.String[] args) [0x010d0] in /Users/builder/data/lanes/5944/64fece5f/source/xamarin-macios/tools/mmp/driver.cs:523
at Xamarin.Bundler.Driver.Main (System.String[] args) [0x00015] in /Users/builder/data/lanes/5944/64fece5f/source/xamarin-macios/tools/mmp/driver.cs:220
The command exited with code 1.
```

### Environment

https://gist.github.com/chamons/97d5af58d9cbcea25c0abfaa5ad910dc

### Build Logs

https://gist.github.com/chamons/11d2506bf935054bffff273b48fc5a6e

### Example Project (If Possible)

[TestInvalidNativeRefs.zip](https://github.com/xamarin/xamarin-macios/files/2185574/TestInvalidNativeRefs.zip)