Hi Diego,

I’ve never seen this particular problem with lxc. The closest I’ve seen was on one of our instances where I found a lot of stale lxc related data being kept from jobs.

I’m also not completely clear where the rsync process is coming from, but that may be something I’ve just never noticed.

Dave

On 28 Nov 2019, at 12:23, Remi Duraffort remi.duraffort@linaro.org wrote:

Hello Diego,

you are the first one (to my knowledge) to report such issues.

Maybe Dave has already seen such issue?

Rgds

Le mar. 26 nov. 2019 à 12:17, Diego Russo <Diego.Russo@arm.com mailto:Diego.Russo@arm.com> a écrit : Hello,

Sometimes LAVA slaves end up in a funky state where all jobs will be failing for the lxc-creation. This is an just a section of the pstree but actually the machine is not doing anything. No CPU/IO/memory usage at all.

|-lxc-ubuntu,26488 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/mbl-rpi3-healthcheck-lxc-86216 --name=mbl-rpi3-healthcheck-lxc-86216 --rootfs=/var/lib/lxc/mbl-rpi3-healthcheck-lxc-86216/rootfs --release xenial ... | `-lxc-ubuntu,26495 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/mbl-rpi3-healthcheck-lxc-86216 --name=mbl-rpi3-healthcheck-lxc-86216 --rootfs=/var/lib/lxc/mbl-rpi3-healthcheck-lxc-86216/rootfs --release xenial ... | `-flock,26496 -x 9 |-lxc-ubuntu,29367 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/cellular-wifi-lxc-85573 --name=cellular-wifi-lxc-85573 --rootfs=/var/lib/lxc/cellular-wifi-lxc-85573/rootfs --release xenial --packages ... | `-lxc-ubuntu,29374 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/cellular-wifi-lxc-85573 --name=cellular-wifi-lxc-85573 --rootfs=/var/lib/lxc/cellular-wifi-lxc-85573/rootfs --release xenial --packages ... | `-rsync,29590 -Ha /var/cache/lxc/xenial/rootfs-amd64/ /var/lib/lxc/cellular-wifi-lxc-85573/rootfs/ | `-rsync,29593 -Ha /var/cache/lxc/xenial/rootfs-amd64/ /var/lib/lxc/cellular-wifi-lxc-85573/rootfs/ | `-rsync,29594 -Ha /var/cache/lxc/xenial/rootfs-amd64/ /var/lib/lxc/cellular-wifi-lxc-85573/rootfs/ |-lxc-ubuntu,30173 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/multi_component-image-update-pelion-lxc-85584 --name=multi_component-image-update-pelion-lxc-85584... | `-lxc-ubuntu,30186 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/multi_component-image-update-pelion-lxc-85584 --name=multi_component-image-update-pelion-lxc-85584... | `-flock,30187 -x 9 |-lxc-ubuntu,30226 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/helloworld-update-lxc-85537 --name=helloworld-update-lxc-85537 --rootfs=/var/lib/lxc/helloworld-update-lxc-85537/rootfs --release xenial --packages ... | `-lxc-ubuntu,30233 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/helloworld-update-lxc-85537 --name=helloworld-update-lxc-85537 --rootfs=/var/lib/lxc/helloworld-update-lxc-85537/rootfs --release xenial --packages ... | `-flock,30234 -x 9 |-lxc-ubuntu,30800 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/rootfs-image-update-mbl-cli-lxc-85588 --name=rootfs-image-update-mbl-cli-lxc-85588 --rootfs=/var/lib/lxc/rootfs-image-update-mbl-cli-lxc-85588/rootfs ... | `-lxc-ubuntu,30807 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/rootfs-image-update-mbl-cli-lxc-85588 --name=rootfs-image-update-mbl-cli-lxc-85588 --rootfs=/var/lib/lxc/rootfs-image-update-mbl-cli-lxc-85588/rootfs ... | `-flock,30808 -x 9 |-lxc-ubuntu,30837 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/mbl-cli-basic-commands-lxc-85540 --name=mbl-cli-basic-commands-lxc-85540 --rootfs=/var/lib/lxc/mbl-cli-basic-commands-lxc-85540/rootfs --release xenial ... | `-lxc-ubuntu,30844 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/mbl-cli-basic-commands-lxc-85540 --name=mbl-cli-basic-commands-lxc-85540 --rootfs=/var/lib/lxc/mbl-cli-basic-commands-lxc-85540/rootfs --release xenial ... | `-flock,30845 -x 9 |-lxc-ubuntu,31697 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/wifi-access-lxc-85591 --name=wifi-access-lxc-85591 --rootfs=/var/lib/lxc/wifi-access-lxc-85591/rootfs --release xenial --packages systemd,systemd-sysv | `-lxc-ubuntu,31704 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/wifi-access-lxc-85591 --name=wifi-access-lxc-85591 --rootfs=/var/lib/lxc/wifi-access-lxc-85591/rootfs --release xenial --packages systemd,systemd-sysv | `-flock,31705 -x 9 |-lxc-ubuntu,31800 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/multi_component-image-update-mbl-cli-lxc-85542 --name=multi_component-image-update-mbl-cli-lxc-85542... | `-lxc-ubuntu,31807 /usr/share/lxc/templates/lxc-ubuntu --path=/var/lib/lxc/multi_component-image-update-mbl-cli-lxc-85542 --name=multi_component-image-update-mbl-cli-lxc-85542... | `-flock,31808 -x 9

I have tens and tens of this processes. LAVA jobs have failed but lxc processes are still there. It seems to be some sort of deadlock in the lxc world. I'm on Debian stretch 9.8. It recently started with no apparent reason.

Regards

-- Diego Russo | Staff Software Engineer | Mbed Linux OS ARM Ltd. CPC1, Capital Park, Cambridge Road, Fulbourn, CB21 5XE, United Kingdom http://www.diegor.co.uk http://www.diegor.co.uk/ - https://os.mbed.com/docs/mbed-linux-os/ https://os.mbed.com/docs/mbed-linux-os/

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.org mailto:Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users https://lists.lavasoftware.org/mailman/listinfo/lava-users

-- Rémi Duraffort LAVA Architect Linaro