Some ideas about improve the 4k video software decoding performance.

I think I should open an issue instead of reply to an issue. So I opened.

I have tested the @lukasf 's work in three days ago. But find its performance is same as the master branch's. So I have to try to optimize it by myself and I found something works.

First, the BufferTime property of MediaStreamSource should set to 0 to disable buffering.
It can extremely significant reduce the memory usage. (From 10GB+ to 700MB, in my 4k 120fps avc 8 bit video.)

Second, I use the performance profiler in visual studio and find the DataWriter object in the UncompressedVideoSampleProvider class eats 10% of my cpu usage. So I have written a class to convert pointer to IBuffer directly.

class BufferReference : public RuntimeClass<
	RuntimeClassFlags<RuntimeClassType::WinRtClassicComMix>,
	abi_IBuffer,
	abi_IBufferByteAccess>
{
private:
	UINT32 m_Capacity;
	UINT32 m_Length;
	byte* m_Pointer;

public:
	virtual ~BufferReference() throw()
	{
	}

	STDMETHODIMP RuntimeClassInitialize(
		byte* Pointer, UINT32 Capacity) throw()
	{
		m_Capacity = Capacity;
		m_Length = Capacity;
		m_Pointer = Pointer;
		return S_OK;
	}

	// IBufferByteAccess::Buffer
	STDMETHODIMP Buffer(byte **value) throw()
	{
		*value = m_Pointer;
		return S_OK;
	}

	// IBuffer::get_Capacity
	STDMETHODIMP get_Capacity(UINT32 *value) throw()
	{
		*value = m_Capacity;
		return S_OK;
	}

	// IBuffer::get_Length
	STDMETHODIMP get_Length(UINT32 *value) throw()
	{
		*value = m_Length;
		return S_OK;
	}

	// IBuffer::put_Length
	STDMETHODIMP put_Length(UINT32 value) throw()
	{
		if (value > m_Capacity)
			return E_INVALIDARG;
		m_Length = value;
		return S_OK;
	}
};

// The M2MakeIBuffer function retrieves the IBuffer object from the provided 
// raw pointer.
//
// Parameters:
//
// Pointer
//     The raw pointer you want to retrieve the IBuffer object.
// Capacity
//     The size of raw pointer you want to retrieve the IBuffer object.
//
// Return value:
//
// If the function succeeds, the return value is the IBuffer object from the 
// provided raw pointer. If the function fails, the return value is nullptr.
//
// Warning: 
// The lifetime of the returned IBuffer object is controlled by the lifetime of
// the raw pointer that's passed to this method. When the raw pointer has been 
// released, the IBuffer object becomes invalid and must not be used.
IBuffer^ M2MakeIBuffer(byte* Pointer, UINT32 Capacity) throw()
{
	IBuffer^ buffer = nullptr;
	
	ComPtr<BufferReference> bufferReference;
	if (SUCCEEDED(MakeAndInitialize<BufferReference>(
		&bufferReference, Pointer, Capacity)))
	{
		buffer = reinterpret_cast<IBuffer^>(bufferReference.Get());
	}

	return buffer;
}

Then I found the maximum fps from 70 to 100. (The goal fps value is 120, just like PotPlayer on my machine.)

I hope someone find a better way to solve, this is the reason why I share it.

I hope it can help you.

Mouri

Start by reviewing the MediaStreamSource BufferTime usage and the UncompressedVideoSampleProvider path described in the issue, then profile the 4K video decoding workload in Visual Studio. Compare memory use and frame rate with the master branch; done means a measured performance improvement toward 120 fps without invalid buffer lifetime behavior.