Fix intermittent "Connection reset by peer" error during port forwarding

https://github.com/rootless-containers/rootlesskit/issues/153Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>