Greetings,
my embedded system is using Mender for over-the-air updates. The way Mender works is that the embedded system needs to have to rootfs partitions, where one is actively being used, and the other is written to when a over-the-air update is initiated. After the new rootfs is written, Mender will change a variable in the bootloader environment and reboot, so that the system boots into the new rootfs. If there is a problem booting into the new rootfs, the bootloader will be set to boot into the old one again. Has anyone here written LAVA tests for systems that use Mender? If so, have you experienced that Mender has complicated the test automation? Any Mender boot test templates would also be appreciated.
On 23 November 2017 at 13:23, Arnstein Kleven arnstein@yoimo.no wrote:
Greetings,
my embedded system is using Mender for over-the-air updates. The way Mender works is that the embedded system needs to have to rootfs partitions, where one is actively being used, and the other is written to when a over-the-air update is initiated. After the new rootfs is written, Mender will change a variable in the bootloader environment and reboot,
Automation generally reacts very badly to unexpected reboots of the device. The boot needs to be expected, i.e. managed in the test job. We've done tests with installers which have similar behaviour. It can work as long as it is LAVA which performs the reboot. Otherwise the prompts get missed, the boot of the new kernel doesn't get tracked correctly and it becomes impossible to do anything useful in the newly booted system.
Writing a variable to the bootloader environment is a significant problem. If the job fails, it is going to be hard to undo that change and get back to a device which can run a test job which doesn't involve OTA. When using the Debian Installer, this is managed by modifying the installation process to *not* install a bootloader / modify the existing bootloader, catching the prompt when the installation is complete and managing the reboot.
In many ways, what you are describing is similar to the secondary media support but is not going to be easy to configure.
so that the system boots into the new rootfs. If there is a problem booting into the new rootfs, the bootloader will be set to boot into the old one again.
How? It would be much better to not have this change in the first place. It is an expected part of OTA but in this case, the automation will need to diverge from the original flow.
Has anyone here written LAVA tests for systems that use Mender? If so, have you experienced that Mender has complicated the test automation? Any Mender boot test templates would also be appreciated.
-- Kind regards, Arnstein Kleven
Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
Mender actually has scripting support, meaning that it can run a script at different points of each stage of the update. For example, one can make a script at the ArtifactReboot_Enter stage (directly before the reboot is executed by Mender), that informs LAVA that a reboot will be performed.
Newer versions of Uboot has variables called bootcount and bootlimit. Mender works by checking that if the bootcount is larger than the bootlimit (meaning that the system is stuck in a boot loop), it will set the variable to which rootfs it should boot to (called mender_boot_part) to the old and working rootfs.
I am new to LAVA and realize that this will definitely complicate the automated testing. However, I will do my best to integrate it and report back to this mailing list, if there is a demand for it.
Arnstein
On 23 November 2017 at 14:37, Neil Williams neil.williams@linaro.org wrote:
On 23 November 2017 at 13:23, Arnstein Kleven arnstein@yoimo.no wrote:
Greetings,
my embedded system is using Mender for over-the-air updates. The way Mender works is that the embedded system needs to have to rootfs partitions, where one is actively being used, and the other is written to when a over-the-air update is initiated. After the new rootfs is written, Mender will change a variable in the bootloader environment and reboot,
Automation generally reacts very badly to unexpected reboots of the device. The boot needs to be expected, i.e. managed in the test job. We've done tests with installers which have similar behaviour. It can work as long as it is LAVA which performs the reboot. Otherwise the prompts get missed, the boot of the new kernel doesn't get tracked correctly and it becomes impossible to do anything useful in the newly booted system.
Writing a variable to the bootloader environment is a significant problem. If the job fails, it is going to be hard to undo that change and get back to a device which can run a test job which doesn't involve OTA. When using the Debian Installer, this is managed by modifying the installation process to *not* install a bootloader / modify the existing bootloader, catching the prompt when the installation is complete and managing the reboot.
In many ways, what you are describing is similar to the secondary media support but is not going to be easy to configure.
so that the system boots into the new rootfs. If there is a problem booting into the new rootfs, the bootloader will be set to boot into the old one again.
How? It would be much better to not have this change in the first place. It is an expected part of OTA but in this case, the automation will need to diverge from the original flow.
Has anyone here written LAVA tests for systems that use Mender? If so, have you experienced that Mender has complicated the test automation? Any Mender boot test templates would also be appreciated.
-- Kind regards, Arnstein Kleven
Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
--
Neil Williams
neil.williams@linaro.org http://www.linux.codehelp.co.uk/
lava-users@lists.lavasoftware.org
-
Arnstein Kleven -
Neil Williams