Stream quality problems with simple h264 encoder

I've written a simple h264 encoder using libva for a screen capture tool. The encoder is set up to create a single IDR frame with only P frames following. I've copied most of the code from the h264 example in libva-utils. I'm facing a problem with image quality of the encoded stream. Errors in the picture seem to accumulate over time and the visual quality deteriorates as more frames are encoded/decoded.

If i understand correctly, the quality should not get worse over time (as long as there are no transmission errors) - the encoder should know about the "reconstruction error" of individual frames and respect them when encoding another. The video I've attached shows a kind of "ghost picture" of past frames which gets more visible over time. I think I might be doing something wrong in my code, however I cannot find out.

The code can be found here:
https://github.com/martin19/snpserver/blob/main/src/stream/video/SnpEncoderVaH264.cpp

A dump of an encoded stream with QP set to constant value of 30 (can be viewed with VLC for example):
https://drive.google.com/file/d/1IdjtmxVxz6KTW8twdLvYodD7vrFhLU-a/view?usp=sharing

(..rgb components are swapped in the stream, so colors are not correct)

Can anyone please help here?