inferbindingsourcesforparameters not working as expected

Hey, I am receiving the following error when I have both [ModelBinder] and [FromBody] tags specified in an action.
"has more than one parameter that was specified or inferred as bound from request body. Only one parameter per action may be bound from body. Inspect the following parameter"

My [ModelBinder] should not be needing the body.
I noticed that the problem goes away when I add (BinderType = typeof(CustomModelBinder))
I can also make it go away when I use https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.mvc.apibehavioroptions.suppressinferbindingsourcesforparameters?view=aspnetcore-5.0#Microsoft_AspNetCore_Mvc_ApiBehaviorOptions_SuppressInferBindingSourcesForParameters

I'm using .NET Core 3.1.

Thanks