I'm having the same issue with this podman version like I have with every 3.1.* version: Rootless podman doesn't work at all anymore:
A simple command, like
podman ps doesn't work - it uses plenty of CPU resources indefinitely and doesn't return a result :(
sudo podman ps works fine. There's no container running on the system.
podman-plugins back to 3.0.1 it works fine again.