On Thu, 31 Jan 2019 at 15:35, Axel Lebourhis axel.lebourhis@linaro.org wrote:

Hi everyone,

I'm writing this email after discussion with Neil. I'm working at NXP and he told me Linaro wanted to run functional tests on imx8m with the new u-boot support. He told me it requires full open access, no license click-through or passwords.

So a bit of background here.

The functional tests I am looking at here would be files in: https://git.lavasoftware.org/lava/functional-tests and the README contains more information: https://git.lavasoftware.org/lava/functional-tests/blob/master/README

Those functional tests are LAVA test jobs which LAVA developers can submit to get assurances that the changes in the master branch of LAVA have not caused a regression in the runtime operations of LAVA. The unit tests can cover a lot of the functionality but some testing of LAVA source code still needs to be done with real devices.

As part of this, Remi has been designing lavafed - Federated LAVA functional testing. https://git.lavasoftware.org/lava/lavafed He will be presenting a lightning talk on lavafed at FOSDEM 2019 in Brussels, this weekend.

lavafed looks like this: https://federation.lavasoftware.org/versions/

The idea is that more and more labs will join into a distributed functional test of LAVA, managed via Docker.

Prior to lavafed, functional tests were submitted either manually or via cron as the lavabuildd user. We have been using functional tests in LAVA for about 3 years now. We would like to get to the point where all new functionality has a functional test.

Philippe Mazet is more qualified to answer this type of question as I only use Android atm. He will follow up the discussion.

Here you have the Yocto source code : https://source.codeaurora.org/external/imx/imx-manifest/tree/README?h=imx-li...

You can get the latest GA release with this :

repo init -u https://source.codeaurora.org/external/imx/imx-manifest -b imx-linux-sumo -m imx-4.14.78-1.0.0_ga.xml

You can build these sources like this :

DISTRO=fsl-imx-wayland MACHINE=imx8mqevk source ./fsl-setup-release.sh -b build

But, this will redirect you to a license click-through.

However, you can bypass the license click-through, like "auto accept it" with this command :

EULA=1 DISTRO=fsl-imx-wayland MACHINE=imx8mqevk source ./fsl-setup-release.sh -b build

We use this bypass to automate builds.

So, let us know if this would be suitable for Linaro's needs.

As it stands, no, those links are not suitable for functional tests but are likely to be useful if and when CI restarts for kernel builds etc. The links above will also be very useful in a README alongside the built files.

Functional tests need static, known working URLs for unchanging kernels, ramdisks, NFS tarballs, DTBs, AOSP images and IoT binaries. We will be using checksums to ensure that these files do not change. The point is to repeatedly run unchanging binaries on unchanging devices, each time the LAVA source code changes. That gives us the assurance we need, by testing one thing at a time. This is for LAVA CI, not Linux CI or OE CI or member company CI. The thing being tested is the LAVA source code, not the devices and not the binaries.

To create functional tests, we need URLs like the ones already in use in the functional tests: https://git.lavasoftware.org/lava/functional-tests/blob/master/release/qemu-... or https://git.lavasoftware.org/lava/functional-tests/blob/master/release/imx7s... or https://git.lavasoftware.org/lava/functional-tests/blob/master/release/hi622... - URLs like:

url: https://files.lavasoftware.org/components/lava/standard/debian/sid/arm64/2/v...

files.lavasoftware.org also has a README: https://files.lavasoftware.org/README

So what we need is the result of the automated builds, at a public location which doesn't use a click-through banner or password, so that we can let LAVA download directly from that URL and put the file onto the device or serve it via TFTP. The zImage files, the foo.dtb files and the something-something.tar.gz NFS root filesystem files. Preferably, all files should come with SHA256 checksums, MD5 can be used too.

We would like to have a script that can rebuild the images - because that helps cover problems when the software in the images/files is itself too old to complete the test and to provide some assurance that people can see how the binary was built to satisfy the licences which apply to the source code used to build the binary in the first place. We need to know that the files are known to work and we only need one set of files for each method of driving the device.

If a suitable LAVA test job is provided too, things will happen more quickly.

The objective here is to get as wide a set of functional tests as we can, across all the devices we can currently access. Then we will increase the contribution into lavafed from other LAVA labs, particularly targeting device-types which we do not have available from the other labs. Once we have a suitably wide set, we can look at deprecating and later pruning some of the boot, deploy or test methods based on the availability of devices which use those methods.

lavafed is in active development and more devices are being added as we develop the support for running test jobs using Docker.

On Fri, 8 Feb 2019 at 08:50, Philippe Mazet philippe.mazet@nxp.com wrote:

Hi Neil,

One additional question:

• Do we have to host the binaries on our servers?

Any public file server will be fine. Via Linaro LDAP, you should have access to people.linaro.org already.

Please make sure that the files are hosted alongside details of all checksums, SHA256 preferably.

• Or can we just provide binaries that you would store on Linaro servers?

I'd prefer not to receive these binaries by email. The whole point is that these are public files, not private to me and not tied up in legal jargon.

The contents of these binaries are expected to be publicly available sources, like kernel mainline, U-Boot or UEFI mainline, OE or Debian sources for the ramdisk/initramfs etc.

Second option would be easier.

Not for me, I'm afraid. I would much prefer simple downloads.

Also, is there any kind of disclaimer/license that can be associated with the binaries distributed on the sites you mentioned in the examples? (like https://files.lavasoftware.org/components/lava/standard/debian/sid/arm64/2/)

No, these need to be public files. The content of the binaries is already going to be under free software licences which already include warranty disclaimers and these are meant to be known working example binaries. I don't understand why any additional licence is required (or applicable).

I think our legal would like to have something that states that the binaries are provided without any kind of warranty, or liability. I will try to confirm this.

The binaries are to contain free software, the details are then according to the licences of the original software.

In context, I think I will have to insist on full detalis of all the sources for all the software contained in these binaries. If the binaries are pulling from public git repos (like Linux, U-Boot mainline etc.) then just list those repos. If pulling from OE or Debian, make it clear which components come from these sources. Anything else will have to have full licence information as free software. We have a responsibility to other users to allow the binaries to be easily rebuilt without reference back to the manufacturer.

The script used to build the binaries will need to be included (this can contain licence information as any regular script can - the licence will need to be open-source as the script will be hosted publicly). Git repos and other source locations listed in the script won't need to be listed elsewhere. The script needs to be able to run without click-through licences or other restrictions. I'm going to need to insist on this script being published alongside the binaries. I am very concerned that these binaries are not being provided with the means to publicly reproduce identical files. The build log will also be required.

Make sure the README is still included - details of the prompts used as with the existing files: https://files.lavasoftware.org/components/lava/standard/debian/sid/arm64/2/d...

I'm not sure why this is proving to be such a problem. Is there proprietary software included in these binaries? Have the necessary changes not been sent upstream? All the upstream projects which I'd expect to be involved are already free software, covered by licences which apply to the binary and source distributions.

All we need is a known working set of files, built from public sources and documented.

Thanks and regards,

Philippe

-----Original Message----- From: Lava-users lava-users-bounces@lists.lavasoftware.org On Behalf Of Neil Williams Sent: Thursday, January 31, 2019 5:07 PM To: Axel Lebourhis axel.lebourhis@linaro.org Cc: lava-users lava-users@lists.lavasoftware.org Subject: Re: [Lava-users] Functional test for imx8m

On Thu, 31 Jan 2019 at 15:35, Axel Lebourhis axel.lebourhis@linaro.org wrote:

...

Hi everyone,

I'm writing this email after discussion with Neil. I'm working at NXP and he told me Linaro wanted to run functional tests on imx8m with the new u-boot support. He told me it requires full open access, no license click-through or passwords.

So a bit of background here.

The functional tests I am looking at here would be files in: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.lavas... and the README contains more information: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.lavas...

Those functional tests are LAVA test jobs which LAVA developers can submit to get assurances that the changes in the master branch of LAVA have not caused a regression in the runtime operations of LAVA. The unit tests can cover a lot of the functionality but some testing of LAVA source code still needs to be done with real devices.

As part of this, Remi has been designing lavafed - Federated LAVA functional testing. https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.lavas... He will be presenting a lightning talk on lavafed at FOSDEM 2019 in Brussels, this weekend.

lavafed looks like this: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffederatio...

The idea is that more and more labs will join into a distributed functional test of LAVA, managed via Docker.

Prior to lavafed, functional tests were submitted either manually or via cron as the lavabuildd user. We have been using functional tests in LAVA for about 3 years now. We would like to get to the point where all new functionality has a functional test.

...

Philippe Mazet is more qualified to answer this type of question as I only use Android atm. He will follow up the discussion.

Here you have the Yocto source code : https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsou rce.codeaurora.org%2Fexternal%2Fimx%2Fimx-manifest%2Ftree%2FREADME%3Fh %3Dimx-linux-sumo&data=02%7C01%7Cphilippe.mazet%40nxp.com%7C43ffc2 4b5ad8449af10508d687964a6a%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0% 7C636845476893813252&sdata=%2FzU%2BbyTpspv2dVT%2BhlIvCOiUMrqDzItvT j0VNQvfP8I%3D&reserved=0

You can get the latest GA release with this :

repo init -u https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsou rce.codeaurora.org%2Fexternal%2Fimx%2Fimx-manifest&data=02%7C01%7C philippe.mazet%40nxp.com%7C43ffc24b5ad8449af10508d687964a6a%7C686ea1d3 bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636845476893813252&sdata=xACmB5 k8DUDqchoUZe9SoCWO9%2Fs9WFG4u4fmy6YD4rw%3D&reserved=0 -b imx-linux-sumo -m imx-4.14.78-1.0.0_ga.xml

You can build these sources like this :

DISTRO=fsl-imx-wayland MACHINE=imx8mqevk source ./fsl-setup-release.sh -b build

But, this will redirect you to a license click-through.

However, you can bypass the license click-through, like "auto accept it" with this command :

EULA=1 DISTRO=fsl-imx-wayland MACHINE=imx8mqevk source ./fsl-setup-release.sh -b build

We use this bypass to automate builds.

So, let us know if this would be suitable for Linaro's needs.

As it stands, no, those links are not suitable for functional tests but are likely to be useful if and when CI restarts for kernel builds etc. The links above will also be very useful in a README alongside the built files.

Functional tests need static, known working URLs for unchanging kernels, ramdisks, NFS tarballs, DTBs, AOSP images and IoT binaries. We will be using checksums to ensure that these files do not change. The point is to repeatedly run unchanging binaries on unchanging devices, each time the LAVA source code changes. That gives us the assurance we need, by testing one thing at a time. This is for LAVA CI, not Linux CI or OE CI or member company CI. The thing being tested is the LAVA source code, not the devices and not the binaries.

To create functional tests, we need URLs like the ones already in use in the functional tests: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.lavas... or https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.lavas... or https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.lavas...

• URLs like:

    url: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffiles.lavasoftware.org%2Fcomponents%2Flava%2Fstandard%2Fdebian%2Fsid%2Farm64%2F2%2Fvmlinuz-4.6.0-1-arm64&data=02%7C01%7Cphilippe.mazet%40nxp.com%7C43ffc24b5ad8449af10508d687964a6a%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636845476893813252&sdata=foqfi56Z%2BZv8UdcpSwpqkFg%2Fk%2F7Dm3N8wM5WExNZfl0%3D&reserved=0
  
  

files.lavasoftware.org also has a README: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffiles.lav...

So what we need is the result of the automated builds, at a public location which doesn't use a click-through banner or password, so that we can let LAVA download directly from that URL and put the file onto the device or serve it via TFTP. The zImage files, the foo.dtb files and the something-something.tar.gz NFS root filesystem files. Preferably, all files should come with SHA256 checksums, MD5 can be used too.

We would like to have a script that can rebuild the images - because that helps cover problems when the software in the images/files is itself too old to complete the test and to provide some assurance that people can see how the binary was built to satisfy the licences which apply to the source code used to build the binary in the first place. We need to know that the files are known to work and we only need one set of files for each method of driving the device.

If a suitable LAVA test job is provided too, things will happen more quickly.

The objective here is to get as wide a set of functional tests as we can, across all the devices we can currently access. Then we will increase the contribution into lavafed from other LAVA labs, particularly targeting device-types which we do not have available from the other labs. Once we have a suitably wide set, we can look at deprecating and later pruning some of the boot, deploy or test methods based on the availability of devices which use those methods.

lavafed is in active development and more devices are being added as we develop the support for running test jobs using Docker.

--

Neil Williams

neil.williams@linaro.org https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linux....

---

Lava-users mailing list Lava-users@lists.lavasoftware.org https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.lav... _______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users