Automatic merge from submit-queue

Predicate cacheing and cleanup

Fix to #31795 

First pass @ cleanup and caching of the CheckServiceAffinity function.  

The cleanup IMO is necessary because the logic around the pod listing and the use of the "implicit selector" (which is reverse engineered to enable the homogenous pod groups).

Should still pass the E2Es.

@timothysc @wojtek-t