Reduce Remove() overhead because we don't have to walk the list to
find an iterator by reference.