Hi folks,

The 2020.04 tag has been pushed to master on git.lavasoftware.org. .deb packages have been built in GitLab CI and are published at

https://apt.lavasoftware.org/release

Docker images for amd64 and arm64 have been built in GitLab CI and are available from

https://hub.lavasoftware.org/

and

https://hub.docker.com/u/lavasoftware

Changes in this release =================

Upgrading =========

Device-type templates ---------------------

Device-type templates as been moved from `/etc/lava-server/dispatcher-config/device-types/` to `/usr/share/lava-server/device-types/`.

Database migrations -------------------

A new `version` field as been added to the `Worker` model. The field will be used to track remove dispatcher version and check for incompatibilities between master and dispatcher.

Device-types ============

New device-types ----------------

New supported devices: * Fixed Virtual Platforms * fsl-ls1043a-rdb * fsl-lx2160a-rdb * fsl-s32v234sbc * imx6q-var-dt6customboard * imx6q-sabresd * imx7d-sdb * imx8mm-ddr4-evk * imx8mm-evk * r8a7795-h3ulcb-kf * sun8i-h3-bananapi-m2-plus

Fixed Virtual Platforms -----------------------

FVP jobs can now be ran in LAVA. LAVA will execute FVP devices inside Docker containers that should be provided by job submitter.

A job definition would be:

```yaml - deploy: to: fvp images: bl1: url: https://example.com/fvp/bl1.bin [...] ramdisk: url: https://example.com/fvp/ramdisk.img

- boot: method: fvp docker: name: "foundation:11.9" local: true image: /opt/model/Foundation_Platformpkg/models/Linux64_GCC-6.4/Foundation_Platform version_string: 'ARM V8 Foundation Platformr0p0 [^\n]+' timeout: minutes: 7 console_string: 'terminal_0: Listening for serial connection on port (?P<PORT>\d+)' arguments: - "--cores=4" - "--no-secure-memory" - "--visualization" - "--gicv3" - "--data={BL1}@0x0" - "--data={FIP}@0x8000000" - "--data={IMAGE}@0x80080000" - "--data={DTB}@0x82000000" - "--data={RAMDISK}@0x84000000" - "--block-device={ROOTFS}" prompts: - 'root@genericarmv8:~#' ```

To use this new device-type, the job definition should define a [deploy](https://docs.lavasoftware.org/lava/actions-deploy.html#to-fvp) and a [boot](https://docs.lavasoftware.org/lava/actions-boot.html#fvp) action.

More information in the [FVP documentation]( https://docs.lavasoftware.org/lava/fvp.html).

UBoot USB Mass Storage ----------------------

LAVA dispatcher will now use `bmaptool` instead of `dd` to flash images. The layout is computed right before the flash.

Docker test action ==================

The docker test action introduced in LAVA [2020.02](https://git.lavasoftware.org/lava/lava/-/wikis/releases/2020.02/) has been improved.

Serial connection to the device -------------------------------

The connection commands are now exposed in the test environment as `LAVA_CONNECTION_COMMAND` and `LAVA_CONNECTION_COMMAND_*`.

For example the given connection commands:

```jinja {% set connection_list = ['uart0', 'uart1'] %} {% set connection_commands = {'uart0': 'telnet localhost 4002', 'uart1': 'telnet 192.168.1.200 8001'} %} ```

Will be exported as:

```shell LAVA_CONNECTION_COMMAND='telnet 192.168.1.200 8001' LAVA_CONNECTION_COMMAND_UART0='telnet localhost 4002' LAVA_CONNECTION_COMMAND_UART1='telnet 192.168.1.200 8001' ```

Waiting for device on USB made optional ---------------------------------------

By default, the docker test action will wait for the device to be connected and exposed to the host via its USB OTG port.

In some use cases, however, you need to e.g. interact with u-boot before a USB OTG port is enabled, so waiting on the device coming up on USB would not work. To avoid waiting for the USB connection, you can specify the `docker.wait.device` parameter set to false:

```yaml actions: # .... - test: docker: image: my-image wait: device: false # .... ```

Device type templates =====================

The default device-type templates has been moved from `/etc/lava-server/dispatcher-config/device-types` to `/usr/share/lava-server/device-types/`.

When rendering device dictionaries, LAVA will use templates in `/etc/lava-server/dispatcher-config/device-types` and then fallback to `/usr/share/lava-server/device-types/`. This mechanism allow admins to override device-type templates.

When using the APIs to override device-type templates, LAVA will write the template into `/etc/lava-server/dispatcher-config/device-types`.

Compression formats ===================

[zstd](https://facebook.github.io/zstd/) as been added to the list of supported compression formats.

In order to use it in a job definition you should use:

```yaml actions: - deploy: rootfs: url: https://example.com/rootfs.ext4.zst compression: zstd ```

The `zstd` package should be installed on the dispatcher. This is already the case for the lava-dispatcher docker image.

Postprocessing images with docker =================================

This release adds support for postprocessing downloaded images using user-provider docker containers. To make use of this feature, you need to use the new ***downloads*** deploy target. Example:

```yaml actions: - deploy: to: downloads images: # ... postprocess: docker: image: my-image steps: - /path/to/my/postprocessing/script ```

The provided docker image will be run with the download directory as current working directory, and the commands listed in `steps` will be executed. Any changes that the commands make to the downloaded images are persisted for later actions, and this includes not only modifying the existing images, but also creating new files in there.

To make use of any files left in the downloads directory, you need a second deploy action, which can refer to files in the downloads directory using the ***downloads://*** pseudo URL scheme. Example:

```yaml - deploy: to: fastboot docker: image: my-adb-fastboot-image images: ptable: url: downloads://ptable-linux-8g.img reboot: hard-reset boot: url: downloads://boot.img reboot: hard-reset system: url: downloads://rpb-console-image-lkft.rootfs.img apply-overlay: true ```

Interactive non-exiting commands ================================

Interactive test received a new field: ```wait_for_prompt```. It defaults to True but can be set to False in the job definition. Idea behind this feature is to allow for non exiting commands to work in non-posix shells. Example can be invoking fastboot in u-boot shell and flashing the board from the subsequent test action ```yaml - test: timeout: minutes: 10 interactive: - name: erase-emmc prompts: ['=> '] script: - command: mmc dev 1 0 name: mmc_dev successes: - message: mmc1(part 0) is current device - command: mmc rescan name: mmc_rescan - command: mmc erase 0 0x400000 name: mmc_erase successes: - message: "4194304 blocks erased: OK" - name: fastboot prompts: ['=> ', '/ # '] script: - command: env default -f -a name: env-default successes: - message: Resetting to default environment - command: setenv partitions $partitions_android name: setenv-partitions - command: fastboot 1 success: - message: "\n" wait_for_prompt: false ```

Boot QEMU from docker image ===========================

We now allow users to provide their own docker image that LAVA will use to start QEMU. This will allow users to use recent QEMU version and to test QEMU itself.

Job definition schema example:

```yaml - boot: method: qemu timeout: minutes: 2 media: tmpfs docker: image: my-qemu-image binary: /usr/bin/qemu-system-x86_64 prompts: - "root@debian:" auto_login: login_prompt: "login:" username: root ```

API ===

New certificate endpoints -------------------------

We've added a support for downloading master certificate and for uploading a worker certificate to master. This will make setting up a remote worker a bit easier for the administrators.

New endpoints available for REST API:

* ```/api/v0.2/workers/${hostname}/certificate/``` * ```/api/v0.2/system/certificate/```

The first one excepts both GET and POST requests so the user can also download the certificate for worker for double checking, while the second one can be used only via GET request and will download the master public key.

New methods available for XMLRPC API:

* #scheduler.workers.get_certificate($hostname) * #scheduler.workers.set_certificate($hostname, $key) * #system.get_master_certificate()

They mirror the REST API calls described above.

Authorization =============

New permissions backend for workers -----------------------------------

The worker model is now protected by the "new" authorization system. You can find settings in the admin section for managing individual workers.

Only `change` permission is used for workers, in order to allow administrators to control who can upload worker certificate, environment and configuration files.

Note that the worker authorization settings will **NOT** affect the device or device type authorization in any way. It's only affecting worker objects and nothing else.

Common device commands now available to test jobs =================================================

LAVA already supported configuring per-device custom commands that can be used from test jobs, like this:

```yaml actions: # ... - command: name: my-custom-command ```

With this release, the following built-in commands can also be used: `pre_power_command`, `pre_os_command`, `power_on`, `power_off`, and `hard_reset`. Although lab admins can configure these commands to be anything, the following table describes their usual semantics:

| Command | Usual meaning | | :------------------ | :------------------------------------ | | `pre_power_command` | Turns USB OTG port ON | | `pre_os_command` | Turns USB OTG port OFF | | `power_on` | Turns power to the board ON | | `power_off` | Turns power to the board OFF | | `hard_reset` | Turns power to the board OFF, then ON |

No extra configuration is required: these commands are already required to be defined for each device, and are now made available to the "command" action without need to explicitly add them to the custom commands configuration.

The changelog is also available in the wiki: https://git.lavasoftware.org/lava/lava/-/wikis/releases/2020.04

Thanks