@imabug: Right, unfortunately we found that 211 does not completely fix #1792623 . It's not a regression compared to 210 (where the CPU/memory graphs pages oopsed everywhere), though. The full fix is in master now.

I just upgraded my system again, which pulled in a new kernel and a new podman, and now it works again.

toolbox-0.0.17-1.fc31.noarch podman-1.7.0-2.fc31.x86_64 crun-0.11-1.fc31.x86_64 kernel-5.4.8-200.fc31.x86_64

So apparently this new crun needs the new podman?

@gscrivano: sudo journalctl --since '2 days ago' | grep mkfifo has no hits, and I didn't see that error message anywhere. I'll test with the previous version in a bit, with ostree that's not entirely trivial.

Slightly more info when trying to start the test container directly with podman:

 podman --log-level=debug start test
DEBU[0000] using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/martin/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/martin/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/martin/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/martin/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/bin/fuse-overlayfs   
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
INFO[0000] running as rootless                          
DEBU[0000] using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/martin/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/martin/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/martin/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/martin/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] overlay: mount_data=lowerdir=/var/home/martin/.local/share/containers/storage/overlay/l/AUK22E4VYYW2GEKXJKNH5ESVJA:/var/home/martin/.local/share/containers/storage/overlay/l/FTUE3LTM4ICA4MKGJ52LZTKLLM,upperdir=/var/home/martin/.local/share/containers/storage/overlay/fe33ed16d2edf5ef2448b0409ca086da52f65edf107cbf460f17012755922dc2/diff,workdir=/var/home/martin/.local/share/containers/storage/overlay/fe33ed16d2edf5ef2448b0409ca086da52f65edf107cbf460f17012755922dc2/work,context="system_u:object_r:container_file_t:s0:c59,c590" 
DEBU[0000] mounted container "6dd6ba8c5dd6129905954c9d651a4264c96b2ce5222af9551a75e95cbdfafa19" at "/var/home/martin/.local/share/containers/storage/overlay/fe33ed16d2edf5ef2448b0409ca086da52f65edf107cbf460f17012755922dc2/merged" 
DEBU[0000] Created root filesystem for container 6dd6ba8c5dd6129905954c9d651a4264c96b2ce5222af9551a75e95cbdfafa19 at /var/home/martin/.local/share/containers/storage/overlay/fe33ed16d2edf5ef2448b0409ca086da52f65edf107cbf460f17012755922dc2/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container 6dd6ba8c5dd6129905954c9d651a4264c96b2ce5222af9551a75e95cbdfafa19 to user.slice:libpod:6dd6ba8c5dd6129905954c9d651a4264c96b2ce5222af9551a75e95cbdfafa19 
DEBU[0000] set root propagation to "rslave"             
DEBU[0000] Created OCI spec for container 6dd6ba8c5dd6129905954c9d651a4264c96b2ce5222af9551a75e95cbdfafa19 at /var/home/martin/.local/share/containers/storage/overlay-containers/6dd6ba8c5dd6129905954c9d651a4264c96b2ce5222af9551a75e95cbdfafa19/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c 6dd6ba8c5dd6129905954c9d651a4264c96b2ce5222af9551a75e95cbdfafa19 -u 6dd6ba8c5dd6129905954c9d651a4264c96b2ce5222af9551a75e95cbdfafa19 -r /usr/bin/crun -b /var/home/martin/.local/share/containers/storage/overlay-containers/6dd6ba8c5dd6129905954c9d651a4264c96b2ce5222af9551a75e95cbdfafa19/userdata -p /run/user/1000/overlay-containers/6dd6ba8c5dd6129905954c9d651a4264c96b2ce5222af9551a75e95cbdfafa19/userdata/pidfile -l k8s-file:/var/home/martin/.local/share/containers/storage/overlay-containers/6dd6ba8c5dd6129905954c9d651a4264c96b2ce5222af9551a75e95cbdfafa19/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog --conmon-pidfile /run/user/1000/overlay-containers/6dd6ba8c5dd6129905954c9d651a4264c96b2ce5222af9551a75e95cbdfafa19/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/martin/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000 --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 6dd6ba8c5dd6129905954c9d651a4264c96b2ce5222af9551a75e95cbdfafa19]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Cleaning up container 6dd6ba8c5dd6129905954c9d651a4264c96b2ce5222af9551a75e95cbdfafa19 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "6dd6ba8c5dd6129905954c9d651a4264c96b2ce5222af9551a75e95cbdfafa19" 
ERRO[0000] unable to start container "test": container create failed (no logs from conmon): EOF 

New creation works:

toolbox create -c test
toolbox enter -c test

But after podman stop test (or rebooting the machine, which is the usual way how to get here):

toolbox --verbose enter -c test
toolbox: running as real user ID 1000
toolbox: resolved absolute path for /bin/toolbox to /usr/bin/toolbox
toolbox: checking if /etc/subgid and /etc/subuid have entries for user martin
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: running on a cgroups v2 host
toolbox: current Podman version is 1.6.2
toolbox: migration not needed: Podman version 1.6.2 is unchanged
toolbox: Fedora generational core is f31
toolbox: base image is fedora-toolbox:31
toolbox: container is test
toolbox: checking if container test exists
toolbox: calling org.freedesktop.Flatpak.SessionHelper.RequestSession
toolbox: starting container test
toolbox: /etc/profile.d/toolbox.sh already mounted in container test
Error: unable to start container "test": container create failed (no logs from conmon): EOF
toolbox: failed to start container test

This seems to break toolbox. Creating a new toolbox works fine (toolbox create), but rebooting and trying to start an existing one (toolbox enter) fails. I'll create a bz with more information shortly.

I created a fresh minimal toolbox with the test case from #1768075, and also my "developer production" toolbox successfully with this update. Thanks!

Tested on my current OSTree, and toolbox is working again. Thanks!

I tested this for #1761765 and it works great, thank you!

@cxhlunar: Well, this is the 196-1 update, and it's been in stable for a month already. The 197-1 update is FEDORA-2019-33e925fa59

Fixed in https://github.com/cockpit-project/cockpit/pull/12236

@cInetbox: Whoops, thanks for the report! This was not intended indeed, so let's not push this to stable. I'll send a PR to drop the dependency on Fedora again.

This update has been unpushed.

@clnetbox: I suppose this got mis-directed here, and you meant that for the cockpit update in FEDORA-2019-33e925fa59

I ran this against the cockpit integration tests, which covers a lot of APIs. Each test checks for new violations, too. I also confirmed that this now works with cockpit-tls. Thanks!

I ran this against the cockpit integration tests, which covers a lot of APIs. Each test checks for new violations, too. I also confirmed that this now works with cockpit-tls. Thanks!

Package installs and works fine.