After the decoder has been initialized and the audio device has been
opened, don't sleep.  The decoder plugin won't do anything special nor
will it care to wake us up for some reason.