Allocating and freeing fences in particular can be expensive, as much as 250 µs
per call (observed on an NVidia machine).