This reverts commit eb0e82a7.
It's causing different deadlocks, notably when a thread tries to
remove a handler inside a handler.