We only care if we have changed the mode and we're changing it back to its
original.  Even if the current mode matches the target mode, we may still
need to release the displays and clear the entry from originalDisplayModes.