Return false on error, not nullptr.

Report unsupported format while opening the filter, not later when the
first conversion takes place.

Rename pcm_convert_domain to pcm_domain.  Move it out so we can use it
without depending on the whole PcmConvert library.