Since we never upload the buffer, and may very well never invalidate the whole
thing, this just results in continually increasing memory consumption.

This is especially a problem for our internal push constant buffers, which are
sysmem buffers that usually are never wholly invalidated, but it could also
potentially be a problem for other buffers.