Enable/disable Formatters per controller or for controllers with produces/consumes

### Is your feature request related to a problem? Please describe.

I need to add XmlSerializerFormatters to my net core 3.1 web api in order to support webhook responses from an external api we're integrating with. When I do this, however, startup attempts to create XmlSerializers for every DTO received and returned by all the actions in all of my controllers, despite only needing XmlSerializers for one action or one controller. Even when using the `[Produces("application/json")` attribute on the non xml controllers, it still attempts to create XmlSerializers for all the DTOs referred to in the controller. While this isn't a huge problem, it is inefficient and it does produce a bunch of
```
warn: Microsoft.AspNetCore.Mvc.Formatters.XmlSerializerOutputFormatter[1]
An error occurred while trying to create an XmlSerializer for the type
```
during startup for DTOs which can't be automatically serialized. I can raise the logging level to error to surpress the warnings, but it doesn't change the fact that startup is still unnecessarily trying to create XmlSerializers.

### Describe the solution you'd like

Either add properties to the MvcXmlOptions to disable/configure this behavior or make IMvcBuilder.AddXmlSerializerFormatters ignore controllers with [Produces()] and [Consumes()] attributes which do not match the corresponding Content-Type.

### Additional References

See this question on stack overflow asked by an additional party with the same issue:
https://stackoverflow.com/questions/60508610/is-it-possible-in-asp-net-core-to-enable-xmlserialization-for-a-single-controlle