Audio codec, priming samples and padding, without container or information from the encoder

How are implementations supposed to handle encoder delay / priming samples / packet padding?

If PCM is encoded with a particular codec, and then decoded, does it roundtrip properly? I don't think it can, because the number of priming samples is not returned by the encoder, so there's no way to mux the file properly to signal this (e.g., using an edit list in an mp4, or any other codec/container pair specific way).

It's possible to make a guess for some codecs (e.g. AAC is frequently 2112 frames), but this isn't ideal (and sometimes it depends on the implementation of a codec).

This is important for accurate synchronization, but a requirement for musical applications (because it breaks sample-accurate looping).