Hi Neil,
One additional question: - Do we have to host the binaries on our servers? - Or can we just provide binaries that you would store on Linaro servers? Second option would be easier.
Also, is there any kind of disclaimer/license that can be associated with the binaries distributed on the sites you mentioned in the examples? (like https://files.lavasoftware.org/components/lava/standard/debian/sid/arm64/2/) I think our legal would like to have something that states that the binaries are provided without any kind of warranty, or liability. I will try to confirm this.
Thanks and regards,
Philippe
-----Original Message----- From: Lava-users lava-users-bounces@lists.lavasoftware.org On Behalf Of Neil Williams Sent: Thursday, January 31, 2019 5:07 PM To: Axel Lebourhis axel.lebourhis@linaro.org Cc: lava-users lava-users@lists.lavasoftware.org Subject: Re: [Lava-users] Functional test for imx8m
On Thu, 31 Jan 2019 at 15:35, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi everyone,
I'm writing this email after discussion with Neil. I'm working at NXP and he told me Linaro wanted to run functional tests on imx8m with the new u-boot support. He told me it requires full open access, no license click-through or passwords.
So a bit of background here.
The functional tests I am looking at here would be files in: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.lavas... and the README contains more information: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.lavas...
Those functional tests are LAVA test jobs which LAVA developers can submit to get assurances that the changes in the master branch of LAVA have not caused a regression in the runtime operations of LAVA. The unit tests can cover a lot of the functionality but some testing of LAVA source code still needs to be done with real devices.
As part of this, Remi has been designing lavafed - Federated LAVA functional testing. https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.lavas... He will be presenting a lightning talk on lavafed at FOSDEM 2019 in Brussels, this weekend.
lavafed looks like this: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffederatio...
The idea is that more and more labs will join into a distributed functional test of LAVA, managed via Docker.
Prior to lavafed, functional tests were submitted either manually or via cron as the lavabuildd user. We have been using functional tests in LAVA for about 3 years now. We would like to get to the point where all new functionality has a functional test.
Philippe Mazet is more qualified to answer this type of question as I only use Android atm. He will follow up the discussion.
Here you have the Yocto source code : https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsou rce.codeaurora.org%2Fexternal%2Fimx%2Fimx-manifest%2Ftree%2FREADME%3Fh %3Dimx-linux-sumo&data=02%7C01%7Cphilippe.mazet%40nxp.com%7C43ffc2 4b5ad8449af10508d687964a6a%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0% 7C636845476893813252&sdata=%2FzU%2BbyTpspv2dVT%2BhlIvCOiUMrqDzItvT j0VNQvfP8I%3D&reserved=0
You can get the latest GA release with this :
repo init -u https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsou rce.codeaurora.org%2Fexternal%2Fimx%2Fimx-manifest&data=02%7C01%7C philippe.mazet%40nxp.com%7C43ffc24b5ad8449af10508d687964a6a%7C686ea1d3 bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636845476893813252&sdata=xACmB5 k8DUDqchoUZe9SoCWO9%2Fs9WFG4u4fmy6YD4rw%3D&reserved=0 -b imx-linux-sumo -m imx-4.14.78-1.0.0_ga.xml
You can build these sources like this :
DISTRO=fsl-imx-wayland MACHINE=imx8mqevk source ./fsl-setup-release.sh -b build
But, this will redirect you to a license click-through.
However, you can bypass the license click-through, like "auto accept it" with this command :
EULA=1 DISTRO=fsl-imx-wayland MACHINE=imx8mqevk source ./fsl-setup-release.sh -b build
We use this bypass to automate builds.
So, let us know if this would be suitable for Linaro's needs.
As it stands, no, those links are not suitable for functional tests but are likely to be useful if and when CI restarts for kernel builds etc. The links above will also be very useful in a README alongside the built files.
Functional tests need static, known working URLs for unchanging kernels, ramdisks, NFS tarballs, DTBs, AOSP images and IoT binaries. We will be using checksums to ensure that these files do not change. The point is to repeatedly run unchanging binaries on unchanging devices, each time the LAVA source code changes. That gives us the assurance we need, by testing one thing at a time. This is for LAVA CI, not Linux CI or OE CI or member company CI. The thing being tested is the LAVA source code, not the devices and not the binaries.
To create functional tests, we need URLs like the ones already in use in the functional tests: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.lavas... or https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.lavas... or https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.lavas... - URLs like:
url: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffiles.lav...
files.lavasoftware.org also has a README: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffiles.lav...
So what we need is the result of the automated builds, at a public location which doesn't use a click-through banner or password, so that we can let LAVA download directly from that URL and put the file onto the device or serve it via TFTP. The zImage files, the foo.dtb files and the something-something.tar.gz NFS root filesystem files. Preferably, all files should come with SHA256 checksums, MD5 can be used too.
We would like to have a script that can rebuild the images - because that helps cover problems when the software in the images/files is itself too old to complete the test and to provide some assurance that people can see how the binary was built to satisfy the licences which apply to the source code used to build the binary in the first place. We need to know that the files are known to work and we only need one set of files for each method of driving the device.
If a suitable LAVA test job is provided too, things will happen more quickly.
The objective here is to get as wide a set of functional tests as we can, across all the devices we can currently access. Then we will increase the contribution into lavafed from other LAVA labs, particularly targeting device-types which we do not have available from the other labs. Once we have a suitably wide set, we can look at deprecating and later pruning some of the boot, deploy or test methods based on the availability of devices which use those methods.
lavafed is in active development and more devices are being added as we develop the support for running test jobs using Docker.