originalDisplayModes should be used when active, empty when inactive.
latentDisplayModes is used when inactive, empty when active.

The count of entries in originalDisplayModes is used to test whether the
process has the displays captured so adding entries when inactive would give
incorrect results.  This could have led us to mistakenly change the display
mode when we don't have the displays captured.