An update: as soon as I kll the rsync processes, all the lxc and flock processes get terminated.

Before the killing I did try to create a lxc container from the command line but it was stuck. It continued after killing the rsync processes.

How can I avoid this situation?

Cheers

-----Original Message----- From: Lava-users lava-users-bounces@lists.lavasoftware.org on behalf of Diego Russo Diego.Russo@arm.com Date: Tuesday, 26 November 2019 at 11:17 To: "lava-users@lists.lavasoftware.org" lava-users@lists.lavasoftware.org Subject: [Lava-users] lxc creation fails with multiple containers

Hello,

Sometimes LAVA slaves end up in a funky state where all jobs will be failing for the lxc-creation. This is an just a section of the pstree but actually the machine is not doing anything. No CPU/IO/memory usage at all.

|-lxc-ubuntu,26488 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/mbl-rpi3-healthcheck-lxc-86216 --name=mbl-rpi3-healthcheck-lxc-86216 --rootfs=/var/lib/lxc/mbl-rpi3-healthcheck-lxc-86216/rootfs --release xenial ... | `-lxc-ubuntu,26495 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/mbl-rpi3-healthcheck-lxc-86216 --name=mbl-rpi3-healthcheck-lxc-86216 --rootfs=/var/lib/lxc/mbl-rpi3-healthcheck-lxc-86216/rootfs --release xenial ... | `-flock,26496 -x 9 |-lxc-ubuntu,29367 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/cellular-wifi-lxc-85573 --name=cellular-wifi-lxc-85573 --rootfs=/var/lib/lxc/cellular-wifi-lxc-85573/rootfs --release xenial --packages ... | `-lxc-ubuntu,29374 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/cellular-wifi-lxc-85573 --name=cellular-wifi-lxc-85573 --rootfs=/var/lib/lxc/cellular-wifi-lxc-85573/rootfs --release xenial --packages ... | `-rsync,29590 -Ha /var/cache/lxc/xenial/rootfs-amd64/ /var/lib/lxc/cellular-wifi-lxc-85573/rootfs/ | `-rsync,29593 -Ha /var/cache/lxc/xenial/rootfs-amd64/ /var/lib/lxc/cellular-wifi-lxc-85573/rootfs/ | `-rsync,29594 -Ha /var/cache/lxc/xenial/rootfs-amd64/ /var/lib/lxc/cellular-wifi-lxc-85573/rootfs/ |-lxc-ubuntu,30173 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/multi_component-image-update-pelion-lxc-85584 --name=multi_component-image-update-pelion-lxc-85584... | `-lxc-ubuntu,30186 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/multi_component-image-update-pelion-lxc-85584 --name=multi_component-image-update-pelion-lxc-85584... | `-flock,30187 -x 9 |-lxc-ubuntu,30226 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/helloworld-update-lxc-85537 --name=helloworld-update-lxc-85537 --rootfs=/var/lib/lxc/helloworld-update-lxc-85537/rootfs --release xenial --packages ... | `-lxc-ubuntu,30233 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/helloworld-update-lxc-85537 --name=helloworld-update-lxc-85537 --rootfs=/var/lib/lxc/helloworld-update-lxc-85537/rootfs --release xenial --packages ... | `-flock,30234 -x 9 |-lxc-ubuntu,30800 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/rootfs-image-update-mbl-cli-lxc-85588 --name=rootfs-image-update-mbl-cli-lxc-85588 --rootfs=/var/lib/lxc/rootfs-image-update-mbl-cli-lxc-85588/rootfs ... | `-lxc-ubuntu,30807 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/rootfs-image-update-mbl-cli-lxc-85588 --name=rootfs-image-update-mbl-cli-lxc-85588 --rootfs=/var/lib/lxc/rootfs-image-update-mbl-cli-lxc-85588/rootfs ... | `-flock,30808 -x 9 |-lxc-ubuntu,30837 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/mbl-cli-basic-commands-lxc-85540 --name=mbl-cli-basic-commands-lxc-85540 --rootfs=/var/lib/lxc/mbl-cli-basic-commands-lxc-85540/rootfs --release xenial ... | `-lxc-ubuntu,30844 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/mbl-cli-basic-commands-lxc-85540 --name=mbl-cli-basic-commands-lxc-85540 --rootfs=/var/lib/lxc/mbl-cli-basic-commands-lxc-85540/rootfs --release xenial ... | `-flock,30845 -x 9 |-lxc-ubuntu,31697 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/wifi-access-lxc-85591 --name=wifi-access-lxc-85591 --rootfs=/var/lib/lxc/wifi-access-lxc-85591/rootfs --release xenial --packages systemd,systemd-sysv | `-lxc-ubuntu,31704 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/wifi-access-lxc-85591 --name=wifi-access-lxc-85591 --rootfs=/var/lib/lxc/wifi-access-lxc-85591/rootfs --release xenial --packages systemd,systemd-sysv | `-flock,31705 -x 9 |-lxc-ubuntu,31800 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/multi_component-image-update-mbl-cli-lxc-85542 --name=multi_component-image-update-mbl-cli-lxc-85542... | `-lxc-ubuntu,31807 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/multi_component-image-update-mbl-cli-lxc-85542 --name=multi_component-image-update-mbl-cli-lxc-85542... | `-flock,31808 -x 9

I have tens and tens of this processes. LAVA jobs have failed but lxc processes are still there. It seems to be some sort of deadlock in the lxc world. I'm on Debian stretch 9.8. It recently started with no apparent reason.

Regards

-- Diego Russo | Staff Software Engineer | Mbed Linux OS ARM Ltd. CPC1, Capital Park, Cambridge Road, Fulbourn, CB21 5XE, United Kingdom http://www.diegor.co.uk - https://os.mbed.com/docs/mbed-linux-os/

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.