Reduce audio resampling noise

Downsampling is better that upsampling, but still it causes a lot of noise. The main reason seem to be because the buffers that get processed (resampled) are very small (~2048 bytes) so the approximations is bad for about 0.1% of the samples. Increasing the decoder output buffer size helps a lot but unfortunately it was not that simple : https://github.com/cbernier2/Transcoder/commit/86986f99a628067e7e0b7a441d7bcb17bed6154a
Is there a simpler way?