Let's refine CoreAudio (Integer Mode, Larger Frame Buffer Size, etc)

Start by reviewing the CoreAudio driver changes described in the issue, including ca_fill_asbd_raw, ca_asbd_equals, ca_asbd_is_better, and ao_read_data_converted, then compare them with the linked commit and working patch. Test the listed integer-mode, frame-buffer, I/O-cycle, Bluetooth, and built-in-device scenarios on relevant macOS hardware; done means the remaining TO-DO items are resolved without the reported noise, distortion, truncation, or sync problems.

mpv version and platform

mpv git version, MacOS 10.9 and newer

Please refer to these files, most recent changes ( https://github.com/macdavis/mpv/commit/5f69382c6393adeeca921c5de2c95982f3c3e2af) and confirmed working one. I am not a programmer, just out of pure interest. Please don't get too surprised If you see some ridiculous codes. All I can do is modifying the existing codes (Quite often, I don't fully understand it). I'm not familiar with using git either. Hope someone who is also interested can rewrite my code, kill the TO-DO list and submit to GitHub in a proper manner. Anyway, after some tests, I manage to make the following improvements on the CoreAudio driver.

Integer Mode ON/OFF and s24 conversion

There are 2 modes in Hog Mode, Float Mode and Integer Mode (OS X 10.6, OS X 10.9 and newer). Float Mode will convert the audio stream to 32-bit floating point before sending to DAC. Integer Mode will send its native format (e.g. 16 or 24-bit integer) directly (w/o format conversion) to DAC. The sound quality of Integer Mode is considered more transparent.

Integer Mode is not available on all devices. Can be checked by the following table.

Mode	Virtual Format	Physical Format
Integer mode	Unmixable 16/24-bit Integer	Unmixable 16/24-bit Integer
Float mode	Mixable 32-bit Floating Point	Mixable 16/24-bit Integer, mixable 32-bit Floating Point

*unmixable/mixable can be checked via kAudioFormatFlagIsNonMixable (ca_print_asbd) or HALLab (free download from Apple)
*Can't check 32-bit integer due to lack of hardware support. It's quite likely that Core Audio doesn't support a physical format of 32-bit integer at all. Some DAC does have a 32 Bit integer physical format. Thus, this Integer Mode patch is needed only if the highest bit hardware supported is 24 Bit (can be checked via kAudioStreamPropertyAvailablePhysicalFormats).

The current mpv git version is actually using Integer Mode. It works in 16-bit integer, but not 24-bit. The reason why 24-bit fails is because we define the format from mpv (32-bit) must match the physical format (24-bit). CoreAudio defines 24-bit audio as "24-bit Alligned high in 32-bit", which means 24-bit is just like 32-bit integer except that we need to specify "mBitsPerChannel" to 24. A patch written by @downthomas (Thank you, downthomas) makes 24-bit Integer Mode work, but broke the non-Exclusive Mode (noise when playing 24-bit audio).

Another way to solve this issue is leaving ca_fill_asbd_raw unchanged, but unchecking a->mBitsPerChannel == b->mBitsPerChannel in function ca_asbd_equals. In this case, mpv's format is 32 Bit aligned high, we sign ASBD to packed 32 Bit, and the physical format is 24 Bit aligned low in 32 Bit. I'm not sure whether these inconsistencies would be a problem, although it sounds okay to me. Personally, I prefer changing ca_fill_asbd_raw and adding a MSB padding function.

CoreAudio tells us exactly what 32/24 Bit format is needed. All we need is doing the conversion accordingly. The hardware format can be retrieved via kAudioStreamPropertyAvailablePhysicalFormats, where some important parameters are,

• mBitsPerChannel can be played around to tell us the highest bit depth (24/32) supported.
• mBytesPerPacket tells you whether is s24 (6) or "s24 in s32" (8).
• asbd->mFormatFlags = kAudioFormatFlagIsAlignedHigh tells us whether is LSB padding (This flag exists) or MSB padding (This flag doesn't exist).

So, the idea is,

1. make a for loop to check kAudioStreamPropertyAvailablePhysicalFormats.
2. if highest_mBitsPerChannel = 32, don't do conversion. //This is a 32 Bit device.
3. if (highest_mBitsPerChannel = 24) & (mBytesPerPacket = 8) & (asbd->mFormatFlags = kAudioFormatFlagIsAlignedHigh), xxxxx. //This is a 24 Bit device with LSB padding in 32 Bit, same as mpv's s32 format.
4. if (highest_mBitsPerChannel = 24) & (mBytesPerPacket = 8) & (NO kAudioFormatFlagIsAlignedHigh flag), xxxxx. //This is a 24 Bit device with MSB padding in 32 Bit.
5. if (highest_mBitsPerChannel = 24) & (mBytesPerPacket = 6) & (asbd->mFormatFlags =kAudioFormatFlagIsPacked), do s24 conversion. //This is a packed s24 device.

Up to now, I managed to set "if case 4 happens, set to standard float mode". The problem is not knowing how to insert the MSB conversion conversion and connect to AO. Thanks to @badgerkin and @wm4, now I use ao_read_data_converted for conversion. Turn out in my case, where device is aligned low, we shall not do the conversion. I'm not sure whether aligned high device needs conversion or not. Since I don't have aligned high, or packed 24 device, both need further testing.

Because mpv doesn't check mFormatFlags (ca_asbd_is_better), in my system, unmixable integer format is always chosen fo both physical and virtual format. Since "our format" is mixable, checking mFormatFlags (kAudioFormatFlagIsNonMixable) will ensure a mixable physical format is selected, which will trigger float conversion (Float Mode) and bypass the "24-bit padded in 32-bit" issue.

Done:

• Make Integer Mode ON/OFF a mpv property and configurable (Integer Mode in ON by default, turn it off by setting coreaudio-integer-mode=no).
• Do MSB padding/s32_to_s24 conversion, if needed.
• Works for both packed and unpacked 24 Bit devices.

TO DO:

• Make downthomas's patch also work in non-Exclusive Mode. Need to add akAudioFormatFlagIsAlignedHigh flag.

• Currently, mpv cannot distinguish between a "true" 32-bit stream and "24-bit padding to 32-bit" stream. Perhaps we can use "bits_per_raw_sample" in AVCodecParameters to determine "24-bit padding to 32-bit" stream, and assign mBitsPerChannel accordingly??? " Our format" is aligned high while the the format of hardware is aligned low, we need to do a format converison. It seems commit 90dd2 added a function to achieve that. But I don't understand it at all.

• If everything works as expected after testing, I shall combine two patches into one.

Increase CoreAudio Frame Buffer Size

Increase CoreAudio Frame Buffer Size (reduce CPU load, from 8.4% to 4.0% when playing 24/96 music on my rMBP) for 2 benefits, energy saving (Apple encourages us to do so, https://developer.apple.com/library/archive/technotes/tn2321/_index.html) and (potentially, suggested by Audirvana and BitPerfect) improving sound quality. On my system (build-in and USB DAC), Frame Buffer Size in float mode can go up to device's maximum value, While that in integer mode can only go to 2043 frames (larger than that will course clicks/distortion).

The actual required Frame Buffer Size range can be different from the device's hardware Frame Size range, e.g., my built-in device's hardware Frame Size range is "14-4096 frames", but the minimum Frame Size to play a Hi-res audio (24/96) is "29 frames" and CoreAudio will automatically increase the size.

The actual Frame Buffer Size is retrieved from "Latency property fsiz".

Setting the Buffer Size beyond the Frame Size range is still playable, as summarised in the following table.

Scenario	Frame Size We Set	Actual Frame Buffer Size
Play 16/44.1 Audio	10	14
Play 16/44.1 Audio	5000	4096
Play 24/96 Audio	14	29

Increasing CoreAudio Frame Buffer Size will also increase audio latency, but WILL NOT effect A/V sync (At least I don't notice any).

Done:

• Make CoreAudio Frame Buffer Size a mpv property and configurable.(The default Frame Buffer Size is increased from 512 to 1024 , other values can be set through coreaudio-buffer-size).
• Automatically set Frame Buffer Size to the maximum value available when Integer Mode is OFF or not available.
• Check the validity of the input Frame Buffer Size. If invalid, set to the closest valid value.

TO DO:

• Dynamically assign the boundary values in OPT_INTRANGE("buffer-size", buffersize, XX, XX, XX) from CA_GET_O(device, kAudioDevicePropertyBufferFrameSizeRange, &value_range) See "Done, point 3".

Reducing IOCycleUsage

According to BitPerfect, reducing IOCycleUsage may improve audio quality.

The default value is 1. Lower than 1 will cause a/v sync problems (The audio will play ahead of video). ONLY use for playing music.

Extreme low value could lead to audio corruption (check whether there is "skipping cycle due to overload" message in Console). Lower CoreAudio Frame Buffer Size requires higher IOCycleUsage. In my system, 0.03 is safe for 24/48, 0.15 for 24/96 with I/O Buffer Size of 2043 Frame; 0.05 for 24/48 with I/O Buffer Size of 1024 Frame.

DONE:

• make CoreAudio IO Cycle Usage a mpv property and configurable (by setting coreaudio-IOCycle-Usage).

Choose a proper format for Bluetooth and Build-in device

My bluetooth device only have 2 physical formats. "8 (aligned high in 16 Bit)/8 1ch" and "32/48 2ch". The virtual format of my bluetooth device is "32-bit float" only. For now, just don't compare "mBitsPerChannel", although it will lead to a "32-bit float" physical format even if a 16-bit file is played. mpv mistakenly take "8 bit aligned high in 16 bit" as true s16, thus, we need to add another criteria, mBytesPerFrame > 3 (mBytesPerFrame of 8 bit is 2, while that of 16 bit is 4). In here, I deliberately set mBytesPerFrame > 5 because I found if not, for build-in device, the case of "32 bit float virtual formant and 16 bit physical format" will create truncation errors and increase the background noise. mBytesPerFrame > 5 simply means that even for true s16, the out physical format will be 32/24bit. Just for test purpose here, the better solution is to separate this to two functions, one for Bluetooth and one for Build-in devices.

Done:

• Probably solve #5238.
• Evaluate "mBytesPerFramel" in a more appropriate way.

TO DO:

• Evaluate "mBitsPerChannel" in a more appropriate way.

As I mentioned earlier, my code definitely needs to be properly rewritten. Those TO-DO Lists need intereactions with mpv core and I have no clues at all. Please feel free to make any suggestions, and if possible, make those lists disappear.