Hello,
Sometimes LAVA slaves end up in a funky state where all jobs will be failing for the lxc-creation. This is an just a section of the pstree but actually the machine is not doing anything. No CPU/IO/memory usage at all.
|-lxc-ubuntu,26488 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/mbl-rpi3-healthcheck-lxc-86216 --name=mbl-rpi3-healthcheck-lxc-86216 --rootfs=/var/lib/lxc/mbl-rpi3-healthcheck-lxc-86216/rootfs --release xenial ... | `-lxc-ubuntu,26495 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/mbl-rpi3-healthcheck-lxc-86216 --name=mbl-rpi3-healthcheck-lxc-86216 --rootfs=/var/lib/lxc/mbl-rpi3-healthcheck-lxc-86216/rootfs --release xenial ... | `-flock,26496 -x 9 |-lxc-ubuntu,29367 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/cellular-wifi-lxc-85573 --name=cellular-wifi-lxc-85573 --rootfs=/var/lib/lxc/cellular-wifi-lxc-85573/rootfs --release xenial --packages ... | `-lxc-ubuntu,29374 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/cellular-wifi-lxc-85573 --name=cellular-wifi-lxc-85573 --rootfs=/var/lib/lxc/cellular-wifi-lxc-85573/rootfs --release xenial --packages ... | `-rsync,29590 -Ha /var/cache/lxc/xenial/rootfs-amd64/ /var/lib/lxc/cellular-wifi-lxc-85573/rootfs/ | `-rsync,29593 -Ha /var/cache/lxc/xenial/rootfs-amd64/ /var/lib/lxc/cellular-wifi-lxc-85573/rootfs/ | `-rsync,29594 -Ha /var/cache/lxc/xenial/rootfs-amd64/ /var/lib/lxc/cellular-wifi-lxc-85573/rootfs/ |-lxc-ubuntu,30173 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/multi_component-image-update-pelion-lxc-85584 --name=multi_component-image-update-pelion-lxc-85584... | `-lxc-ubuntu,30186 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/multi_component-image-update-pelion-lxc-85584 --name=multi_component-image-update-pelion-lxc-85584... | `-flock,30187 -x 9 |-lxc-ubuntu,30226 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/helloworld-update-lxc-85537 --name=helloworld-update-lxc-85537 --rootfs=/var/lib/lxc/helloworld-update-lxc-85537/rootfs --release xenial --packages ... | `-lxc-ubuntu,30233 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/helloworld-update-lxc-85537 --name=helloworld-update-lxc-85537 --rootfs=/var/lib/lxc/helloworld-update-lxc-85537/rootfs --release xenial --packages ... | `-flock,30234 -x 9 |-lxc-ubuntu,30800 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/rootfs-image-update-mbl-cli-lxc-85588 --name=rootfs-image-update-mbl-cli-lxc-85588 --rootfs=/var/lib/lxc/rootfs-image-update-mbl-cli-lxc-85588/rootfs ... | `-lxc-ubuntu,30807 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/rootfs-image-update-mbl-cli-lxc-85588 --name=rootfs-image-update-mbl-cli-lxc-85588 --rootfs=/var/lib/lxc/rootfs-image-update-mbl-cli-lxc-85588/rootfs ... | `-flock,30808 -x 9 |-lxc-ubuntu,30837 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/mbl-cli-basic-commands-lxc-85540 --name=mbl-cli-basic-commands-lxc-85540 --rootfs=/var/lib/lxc/mbl-cli-basic-commands-lxc-85540/rootfs --release xenial ... | `-lxc-ubuntu,30844 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/mbl-cli-basic-commands-lxc-85540 --name=mbl-cli-basic-commands-lxc-85540 --rootfs=/var/lib/lxc/mbl-cli-basic-commands-lxc-85540/rootfs --release xenial ... | `-flock,30845 -x 9 |-lxc-ubuntu,31697 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/wifi-access-lxc-85591 --name=wifi-access-lxc-85591 --rootfs=/var/lib/lxc/wifi-access-lxc-85591/rootfs --release xenial --packages systemd,systemd-sysv | `-lxc-ubuntu,31704 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/wifi-access-lxc-85591 --name=wifi-access-lxc-85591 --rootfs=/var/lib/lxc/wifi-access-lxc-85591/rootfs --release xenial --packages systemd,systemd-sysv | `-flock,31705 -x 9 |-lxc-ubuntu,31800 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/multi_component-image-update-mbl-cli-lxc-85542 --name=multi_component-image-update-mbl-cli-lxc-85542... | `-lxc-ubuntu,31807 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/multi_component-image-update-mbl-cli-lxc-85542 --name=multi_component-image-update-mbl-cli-lxc-85542... | `-flock,31808 -x 9
I have tens and tens of this processes. LAVA jobs have failed but lxc processes are still there. It seems to be some sort of deadlock in the lxc world. I'm on Debian stretch 9.8. It recently started with no apparent reason.
Regards
-- Diego Russo | Staff Software Engineer | Mbed Linux OS ARM Ltd. CPC1, Capital Park, Cambridge Road, Fulbourn, CB21 5XE, United Kingdom http://www.diegor.co.uk - https://os.mbed.com/docs/mbed-linux-os/
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.