We should swap min & max only when both limits are being
set. Otherwise limit that being set should invalidate other limit
(remove it).