Publishing single file WPF application breaks if app.config is used

If you publish a WPF application in .NET 5, and this application is using an App.config, the application crashes when the framework is trying to find this file, even though the file is part of the published output.
The following settings are configured in project:
```

win-x64
true
true
true
true
none
false

```
And when publishing the project with `dotnet publish -c Release`, then the output looks good:
Correct name of the config file is there.

However, when the app is trying to access any setting in this file with ConfigurationManager, the app crashes:

```
Application: WpfNet5App.exe
CoreCLR Version: 5.0.20.51904
.NET Version: 5.0.0
Description: The process was terminated due to an unhandled exception.
Exception Info: System.IO.FileNotFoundException: The system cannot find the file specified. (0x80070002)
at System.Reflection.RuntimeModule.GetFullyQualifiedName()
at System.Reflection.RuntimeModule.get_Name()
at System.Configuration.ClientConfigPaths..ctor(String exePath, Boolean includeUserConfig)
at System.Configuration.ClientConfigPaths.GetPaths(String exePath, Boolean includeUserConfig)
at System.Configuration.Internal.ConfigurationManagerInternal.System.Configuration.Internal.IConfigurationManagerInternal.get_ExeProductName()
at System.Configuration.ApplicationSettingsBase.get_Initializer()
at System.Configuration.ApplicationSettingsBase.CreateSetting(PropertyInfo propertyInfo)
at System.Configuration.ApplicationSettingsBase.EnsureInitialized()
at System.Configuration.ApplicationSettingsBase.get_Properties()
at System.Configuration.SettingsBase.GetPropertyValueByName(String propertyName)
at System.Configuration.SettingsBase.get_Item(String propertyName)
at System.Configuration.ApplicationSettingsBase.GetPropertyValue(String propertyName)
at System.Configuration.ApplicationSettingsBase.get_Item(String propertyName)
at WpfNet5App.Properties.Settings.get_HelloMessage()
at WpfNet5App.MainWindow.Button_Click(Object sender, RoutedEventArgs e)
```

Attaching the super-simple example.
[WpfNet5App.zip](https://github.com/dotnet/core/files/5630310/WpfNet5App.zip)