This is a pretty naive implementation, should that become a performance
problem it's easy enough to speed up with a search tree of some kind.