On Fri, Aug 14, 2020 at 05:09:37PM +0000, Stephen Lawrence wrote:
-----Original Message----- From: Stephen Lawrence Sent: 12 August 2020 18:03 To: Antonio Terceiro antonio.terceiro@linaro.org Cc: lava-users@lists.lavasoftware.org Subject: RE: fastboot reboot-bootloader from docker test shell (was "fastboot in docker support")
-----Original Message----- From: Antonio Terceiro antonio.terceiro@linaro.org Sent: 12 August 2020 14:59 To: Stephen Lawrence stephen.lawrence@renesas.com Cc: lava-users@lists.lavasoftware.org Subject: Re: fastboot reboot-bootloader from docker test shell (was "fastboot in
docker
support")
[snip]
It should be enough. see if there are any errors in the udev logs. e.g. if lava-dispatcher-host fails for some reason there will be a correspoding line there (but no output unfortunately as udev does not capture it, only the failure exit code)
Thanks for the hints. Nothing in the host logs from lava-dispatcher-host via journalctl. In the worker container there is no systemd and in the lava-slave /var/log I don't see anything.
At this current moment it feels more a fundamental lava worker in docker issue given the lack of udev events being reported by udevadm monitor in the worker container. That the udev discovery code in lava-dispatcher-host has no chance if there are no events. Unless something about the container environment means I can not take the udevadm reporting at face value.
btw just a couple of additional comments about my post earlier today. I'm really trying not to be that new guy to a list who just spams with their issues and 'fix this next one' one after another. I did not mean that the developers should be maintaining some nice document set for running LAVA in docker.
I just meant two main points. Firstly, how the devs thought udev should work in such a lava environment - if its been considered. Secondly, that the wider community can collaborate on some patterns as to what needs to be shared into a container for certain LAVA services to work. Of course that can be codified where sensible in the docker base image or downstream projects like lava-docker. Web searches about enabling udev in docker containers returns very little outside some hacks.
It looks like I have finally tracked this down to its cause. Looking back at my notes when I was debugging lava_dispatcher/utils/udev.py it was looking like the lack of udev events in the worker container might be a red herring. So I went back to the lava-dispatcher-host source and the udev rules it appears to rely on.
lava-dispatcher-host was installed but in the worker container with the rest of the worker. Not receiving udev events in the worker container means that the rule there never executes. At the same time the host does receive the udev events on DUT reboot but there is nothing to communicate them to the worker container.
lava-dispatcher-host needs to be installed and configured (i.e. udev rules in place) on the host system for it to work, exactly because udev events are not (and probably should not be) available inside the container.
This then goes back to the first question I raised yesterday about how the lava devs see udev discovery commonly working when LAVA is containerised. For example, whether it happens on the host and is somehow communicated into the worker container, or whether 'full' udev is required in the worker container or whether in some cases a different setup is possible. I'll report on the last choice below.
Running the dispatcher in a container and still being able to run jobs inside containers is still WIP from our PoV, I am only now starting to look at it.