The current behavior is that if audio is behind, all future audio-samples are skipped, resulting in instantly getting correct sync, but also a harsh sound behavior.

Have the application treat the "extra audio" a bit better than throwing it out.
On each frame: If there is extra audio, and it's more than N packets too many, rescale it to some fraction of the original length.
Some experimentation is required, but this should make the audio lag gradually dissapear, over some time (seconds), without the audio artifacts resulting from skipping.