Swagger Serializer Settings

**Describe the issue**
I cannot find anywhere that allows me to set the settings or even which serializer swagger uses to serialize my objects. I utilize custom serializer settings (Newtonsoft) that send my objects differently depending on the destination. In my application these settings change the output or even hide fields.

For example, to a database, I may want ENUMs to use numbers for storage, but for APIs (and swagger) I would want strings. Or I may want some fields exposed for different levels of authentication for a publicly available API. Decorating functions won't work because the I need the behavior to change at runtime.

I've seen the following as recommendation in the issues, but this seems to have no effect whatsoever. Sadly I've not been able to find anything else in the documentation.
```
var host = new HostBuilder()
.ConfigureFunctionsWebApplication(builder =>
{
builder.UseNewtonsoftJson();
})
.Build();
host.Run();
```

**The Question**
What serializer is used by swagger/openapi?
How can one alter the settings and add customer converters?