Hi folks,
The 2020.05 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=================
Upgrading
=========
Database migrations
-------------------
A new `master_version_notified` field has been added to the `Worker` model.
The field will be used to know if admins where notified of the version
mismatch between master and dispatchers.
Device-type templates
---------------------
The post-installation script will look at the device-types templates in
`/etc/lava-server/dispatcher-config/device-types/` and remove the templates
that are already present in `/usr/share/lava-server/device-types/` (and
identical).
Device-types
============
New device-types
----------------
New supported devices:
* imx6q-sabreauto
* imx8dxl-evk
* imx8mp-evk
* imx8qm-mek
* imx8qxp-mek
* ls1012ardb
* ls1028ardb
* ls1046ardb
* ls1088ardb
* fsl-s32v234sbc (and s32v234-evb)
* mt8173-elm-hana
U-Boot
------
The `uboot_error_messages` variable allows device-type templates to set
their own list of u-boot error messages as required if the default set is
not appropriate. For example it may be required that one of the default
error messages is ignored.
```jinja
{% set uboot_error_messages = [
'Resetting CPU',
'Must RESET board to recover',
'TIMEOUT']
%}
```
The `uboot_extra_error_message` variable can still be used in conjunction
with `uboot_error_messages`.
Minnowboard
-----------
Replace `boot_message` by `kernel_start_message` as the latter has been
deprecated for a long time.
nfs boot commands
-----------------
Drop "intr" mount option that has been deprecated in 2.6.25. This is the
default value since 2.6.25.
OpenOCD
-------
Open serial connection prior to invoking OpenOCD in OpenOCD boot method
The serial connection should be opened prior to invoking OpenOCD. This
fixes an issue where on some devices verbose serial output is truncated
when the data size exceeds buffering in the firmware.
Download
========
When downloading artifacts fail for network issues, LAVA `dispatcher` will
retry up to 15 times over 10 minutes.
These retries will fix some intermittent failures.
API
===
New system endpoints
--------------------
We've added a support for getting the system version and current user
New endpoints available for REST API:
* `/api/v0.2/system/version/`
* `/api/v0.2/system/whoami/`
New test endpoints
------------------
The endpoint will allow to access the tests for a given job at:
* `/api/v0.2/jobs/<job_id>/tests/`
The results are also available at:
* `/api/v0.2/jobs/<job_id>/suites/<suite_id>/tests/`
docker
======
Site
-----
When starting `lava-server`, you can set the `Site` by setting the
`LAVA_SITE` environment variable.
Superuser
---------
When starting `lava-server`, the entrypoint can create a super user for
you. Just set `LAVA_ADMIN_USERNAME` and `LAVA_ADMIN_PASSWORD` environment
variable.
Interactive tests and multinode
===============================
Implement multinode synchronization and `delay` primitives.
Introduce `delay` primitive to wait a given number of seconds (incl.
fractional) and `lava-send`, `lava-wait`, `lava-wait-all`, `lava-sync` for
multinode synchronization.
Syntax for multinode primitives follows one used in test-definitions, i.e.
single-line based.
A job definition would look like:
```yaml
- test:
role: [server]
interactive:
- name: boot
prompts: ["/ #"]
echo: discard
script:
- command: ifconfig
name: result
successes:
- message: "inet addr:(?P<ip>\\d+\\.\\d+\\.\\d+\\.\\d+)"
- lava-send: booted ipaddr={ip}
- test:
role: [client]
interactive:
- name: boot
prompts: ["/ #"]
echo: discard
script:
- delay: 5
- lava-send: booted
- lava-wait-all: booted
- command: 'echo "Other side has IP: {ipaddr}"'
```
In this example, LAVA will capture the IP in the success message on the
server and use the value in the client command.
LAVA settings
=============
In previous LAVA versions, the settings are stored in:
* `/etc/lava-server/instance.conf`: database settings
* `/etc/lava-server/settings.conf`: global settings (json)
* `/etc/lava-server/secret_key.conf`: secret key created on the fly
In order to make admin task easier, the settings are now stored in yaml
files under `/etc/lava-server/settings.d/`.
The legacy configuration files will be loaded first and then the files in
`/etc/lava-server/settings.d/` in alphabetical order.
Upgrade notification
====================
When `lava-master` restarts, it will check that the remote dispatchers are
running the same version. If that's not the case, `lava-master` will send a
mail to each worker admin.
This feature can be activated in the settings:
```yaml
MASTER_UPGRADE_NOTIFY: true
```
Test job log
============
Prior to this release, LAVA would always store test job logs on the
filesystem in
`/var/lib/lava-server/default/media/job-output/<year>/<month>/<day>/<id>`.
Starting from this release, admins can configure the logger backend to
either the filesystem (default) or [mongodb](https://www.mongodb.com/).
In order to use Mongodb, admin should install `python3-pymongo` and update
the settings:
```yaml
LAVA_LOG_BACKEND: "lava_scheduler_app.logutils.LogsMongo"
MONGO_DB_URI: "mongodb://<username>:<password>@localhost:27017/"
```
The mongodb support is currently in Beta mainly because performances can
still be improved.
Thanks
--
Rémi Duraffort
LAVA Architect
Linaro
Hi folks,
The 2020.04 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=================
Upgrading
=========
Device-type templates
---------------------
Device-type templates as been moved from
`/etc/lava-server/dispatcher-config/device-types/` to
`/usr/share/lava-server/device-types/`.
Database migrations
-------------------
A new `version` field as been added to the `Worker` model. The field will
be used to track remove dispatcher version and check for incompatibilities
between master and dispatcher.
Device-types
============
New device-types
----------------
New supported devices:
* Fixed Virtual Platforms
* fsl-ls1043a-rdb
* fsl-lx2160a-rdb
* fsl-s32v234sbc
* imx6q-var-dt6customboard
* imx6q-sabresd
* imx7d-sdb
* imx8mm-ddr4-evk
* imx8mm-evk
* r8a7795-h3ulcb-kf
* sun8i-h3-bananapi-m2-plus
Fixed Virtual Platforms
-----------------------
FVP jobs can now be ran in LAVA. LAVA will execute FVP devices inside Docker
containers that should be provided by job submitter.
A job definition would be:
```yaml
- deploy:
to: fvp
images:
bl1:
url: https://example.com/fvp/bl1.bin
[...]
ramdisk:
url: https://example.com/fvp/ramdisk.img
- boot:
method: fvp
docker:
name: "foundation:11.9"
local: true
image:
/opt/model/Foundation_Platformpkg/models/Linux64_GCC-6.4/Foundation_Platform
version_string: 'ARM V8 Foundation Platformr0p0 [^\\n]+'
timeout:
minutes: 7
console_string: 'terminal_0: Listening for serial connection on port
(?P<PORT>\d+)'
arguments:
- "--cores=4"
- "--no-secure-memory"
- "--visualization"
- "--gicv3"
- "--data={BL1}@0x0"
- "--data={FIP}@0x8000000"
- "--data={IMAGE}@0x80080000"
- "--data={DTB}@0x82000000"
- "--data={RAMDISK}@0x84000000"
- "--block-device={ROOTFS}"
prompts:
- 'root@genericarmv8:~#'
```
To use this new device-type, the job definition should define a
[deploy](https://docs.lavasoftware.org/lava/actions-deploy.html#to-fvp)
and a
[boot](https://docs.lavasoftware.org/lava/actions-boot.html#fvp)
action.
More information in the [FVP documentation](
https://docs.lavasoftware.org/lava/fvp.html).
UBoot USB Mass Storage
----------------------
LAVA dispatcher will now use `bmaptool` instead of `dd` to flash images. The
layout is computed right before the flash.
Docker test action
==================
The docker test action introduced in LAVA
[2020.02](https://git.lavasoftware.org/lava/lava/-/wikis/releases/2020.02/)
has
been improved.
Serial connection to the device
-------------------------------
The connection commands are now exposed in the test environment as
`LAVA_CONNECTION_COMMAND` and `LAVA_CONNECTION_COMMAND_*`.
For example the given connection commands:
```jinja
{% set connection_list = ['uart0', 'uart1'] %}
{% set connection_commands = {'uart0': 'telnet localhost 4002', 'uart1':
'telnet 192.168.1.200 8001'} %}
```
Will be exported as:
```shell
LAVA_CONNECTION_COMMAND='telnet 192.168.1.200 8001'
LAVA_CONNECTION_COMMAND_UART0='telnet localhost 4002'
LAVA_CONNECTION_COMMAND_UART1='telnet 192.168.1.200 8001'
```
Waiting for device on USB made optional
---------------------------------------
By default, the docker test action will wait for the device to be connected
and exposed to the host via its USB OTG port.
In some use cases, however, you need to e.g. interact with u-boot before a
USB OTG port is enabled, so waiting on the device coming up on USB would
not work. To avoid waiting for the USB connection, you can specify the
`docker.wait.device` parameter set to false:
```yaml
actions:
# ....
- test:
docker:
image: my-image
wait:
device: false
# ....
```
Device type templates
=====================
The default device-type templates has been moved from
`/etc/lava-server/dispatcher-config/device-types` to
`/usr/share/lava-server/device-types/`.
When rendering device dictionaries, LAVA will use templates in
`/etc/lava-server/dispatcher-config/device-types` and then fallback to
`/usr/share/lava-server/device-types/`. This mechanism allow admins to
override
device-type templates.
When using the APIs to override device-type templates, LAVA will write the
template into `/etc/lava-server/dispatcher-config/device-types`.
Compression formats
===================
[zstd](https://facebook.github.io/zstd/) as been added to the list of
supported compression formats.
In order to use it in a job definition you should use:
```yaml
actions:
- deploy:
rootfs:
url: https://example.com/rootfs.ext4.zst
compression: zstd
```
The `zstd` package should be installed on the dispatcher. This is already
the case for the lava-dispatcher docker image.
Postprocessing images with docker
=================================
This release adds support for postprocessing downloaded images using
user-provider docker containers. To make use of this feature, you need to
use the new ***downloads*** deploy target. Example:
```yaml
actions:
- deploy:
to: downloads
images:
# ...
postprocess:
docker:
image: my-image
steps:
- /path/to/my/postprocessing/script
```
The provided docker image will be run with the download directory as
current working directory, and the commands listed in `steps` will be
executed. Any changes that the commands make to the downloaded images are
persisted for later actions, and this includes not only modifying the
existing images, but also creating new files in there.
To make use of any files left in the downloads directory, you need a second
deploy action, which can refer to files in the downloads directory using
the ***downloads://*** pseudo URL scheme. Example:
```yaml
- deploy:
to: fastboot
docker:
image: my-adb-fastboot-image
images:
ptable:
url: downloads://ptable-linux-8g.img
reboot: hard-reset
boot:
url: downloads://boot.img
reboot: hard-reset
system:
url: downloads://rpb-console-image-lkft.rootfs.img
apply-overlay: true
```
Interactive non-exiting commands
================================
Interactive test received a new field: ```wait_for_prompt```. It defaults
to True but can be set to False in the job definition. Idea behind this
feature is to allow for non exiting commands to work in non-posix shells.
Example can be invoking fastboot in u-boot shell and flashing the board
from the subsequent test action
```yaml
- test:
timeout:
minutes: 10
interactive:
- name: erase-emmc
prompts: ['=> ']
script:
- command: mmc dev 1 0
name: mmc_dev
successes:
- message: mmc1(part 0) is current device
- command: mmc rescan
name: mmc_rescan
- command: mmc erase 0 0x400000
name: mmc_erase
successes:
- message: "4194304 blocks erased: OK"
- name: fastboot
prompts: ['=> ', '/ # ']
script:
- command: env default -f -a
name: env-default
successes:
- message: Resetting to default environment
- command: setenv partitions $partitions_android
name: setenv-partitions
- command: fastboot 1
success:
- message: "\n"
wait_for_prompt: false
```
Boot QEMU from docker image
===========================
We now allow users to provide their own docker image that LAVA will use to
start QEMU.
This will allow users to use recent QEMU version and to test QEMU itself.
Job definition schema example:
```yaml
- boot:
method: qemu
timeout:
minutes: 2
media: tmpfs
docker:
image: my-qemu-image
binary: /usr/bin/qemu-system-x86_64
prompts:
- "root@debian:"
auto_login:
login_prompt: "login:"
username: root
```
API
===
New certificate endpoints
-------------------------
We've added a support for downloading master certificate and for uploading a
worker certificate to master. This will make setting up a remote worker a
bit
easier for the administrators.
New endpoints available for REST API:
* ```/api/v0.2/workers/${hostname}/certificate/```
* ```/api/v0.2/system/certificate/```
The first one excepts both GET and POST requests so the user can also
download
the certificate for worker for double checking, while the second one can be
used only via GET request and will download the master public key.
New methods available for XMLRPC API:
* #scheduler.workers.get_certificate($hostname)
* #scheduler.workers.set_certificate($hostname, $key)
* #system.get_master_certificate()
They mirror the REST API calls described above.
Authorization
=============
New permissions backend for workers
-----------------------------------
The worker model is now protected by the "new" authorization system. You can
find settings in the admin section for managing individual workers.
Only `change` permission is used for workers, in order to allow
administrators
to control who can upload worker certificate, environment and
configuration files.
Note that the worker authorization settings will **NOT** affect the device
or
device type authorization in any way. It's only affecting worker objects and
nothing else.
Common device commands now available to test jobs
=================================================
LAVA already supported configuring per-device custom commands that can be
used
from test jobs, like this:
```yaml
actions:
# ...
- command:
name: my-custom-command
```
With this release, the following built-in commands can also be used:
`pre_power_command`,
`pre_os_command`, `power_on`, `power_off`, and `hard_reset`. Although lab
admins can
configure these commands to be anything, the following table describes
their usual
semantics:
| Command | Usual meaning |
| :------------------ | :------------------------------------ |
| `pre_power_command` | Turns USB OTG port ON |
| `pre_os_command` | Turns USB OTG port OFF |
| `power_on` | Turns power to the board ON |
| `power_off` | Turns power to the board OFF |
| `hard_reset` | Turns power to the board OFF, then ON |
No extra configuration is required: these commands are already required to
be defined
for each device, and are now made available to the "command" action without
need to
explicitly add them to the custom commands configuration.
The changelog is also available in the wiki:
https://git.lavasoftware.org/lava/lava/-/wikis/releases/2020.04
Thanks
--
Rémi Duraffort
LAVA Architect
Linaro
Hello,
As you know, we are trying to release a new version of LAVA every month.
We are delaying this month release (2020.03) due to some issues and crashes
that we found in the last days.
Issues where found on staging.vlo, lavafed and meta-lava, proving that the
CI is working.
Where are currently working on fixing the issues and proving that LAVA is
ready for release.
The next release should be ready for the end of next week and would be
called 2020.04.
Rgds
--
Rémi Duraffort
LAVA Architect
Linaro
Hi folks,
The 2020.02 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=================
Upgrading
=========
Debian stretch
--------------
The support for Debian Stretch as been dropped. The minimal supported
version is now Debian Buster.
Security issues
---------------
Two security issues have been discovered and fixed in LAVA:
* remote code execution on the dispatcher (as root)
* overriding any directory on the dispatcher filesystem
Every version of LAVA (since 2014) are affected. However, only
authenticated users can exploit these issues.
We strongly advise to upgrade your instances to LAVA 2020.02.
Device-types
============
New device-types
----------------
New supported devices:
* bcm2711-rpi-4-b
* meson-sm1-khadas-vim3l
* musca-a
* musca-b
* rk3288-miqi
* sun50i-h5-nanopi-neo-plus2
docker
------
ADmins can add any extra argument to the docker DUT by setting
`docker_extra_arguments` in the device dictionary:
```jinja
{% set docker_extra_arguments = ["--ip", "10.0.1.3"] %}
```
MPS2
----
Allow to use the `flasher` deploy method.
Musca
-----
Add support for Musca boards.
```yaml
actions:
- deploy:
to: musca
images:
test_binary:
url: https://community.arm.com/.../MuscaBlinky_5F00_v002.hex
- boot:
method: musca
```
UUU deploy and boot
--------------------
[UUU](https://github.com/NXPmicro/mfgtools) is NXP's flashing tool for IMX
platforms.
UUU allows to intercept IMX ROM code, and download images on target by
using SDP (Serial Download Protocol).
IMX platforms are booting in SDP mode based on dip switches, but also falls
back in SDP mode when no valid bootloader is found in the selected boot
media (eg: SD card, or eMMC).
```yaml
actions:
- deploy:
to: uuu
images:
boot:
url: https://example.com/initramfs-genericarmv7a.cpio.gz
compression: gz
timeout:
minutes: 2
- boot:
method: uuu
commands:
- uuu: -b sd {boot}
timeout:
minutes: 4
```
vexpress
--------
The u-boot commands are now send one by one. The use of environment
variable to store boot commands has also been removed in favor of executing
the commands immediately.
This will allow to detect error early in the job.
Docker test action
==================
LAVA dispatcher can now run a test action in a docker container running on
the host. This feature can be used to run android tests from a user-defined
docker container against the DUT.
```yaml
- test:
timeout:
minutes: 10
docker:
image: adb-fastboot
definitions:
- repository:
metadata:
format: Lava-Test Test Definition 1.0
name: smoke-adb-tests
description: "Simple adb tests"
run:
steps:
- date
- hostname
- adb devices
- adb shell date
- adb shell hostname
from: inline
path: inline-smoke-test
name: docker-test
```
Image modification
==================
LAVA dispatcher can now append user-defined overlays to images. For
example, users can now ask LAVA to add some modules to the rootfs.
Currently, the image to update should be either a `cpio` (newc) or an
`ext4` image. The overlays should be tar archives.
The job definition would look like this:
```yaml
rootfs:
url: http://example.com/rootfs.ext4.xz
compression: xz
format: ext4
overlays:
modules:
url: http://example.com/modules.tar.gz
compression: gz
format: tar
path: /
```
Dispatcher configuration
========================
Allow to set `http`, `nfs` and `tftp` dispatcher ip in the dispatcher
configuration.
Edit `/etc/lava-server/dispatcher.d/<hostname>/dispatcher.yaml`:
```yaml
dispatcher_http_ip: <dispatcher-http-ip>:<port>
dispatcher_nfs_ip: <dispatcher-nfs-ip>:<port>
dispatcher_tftp_ip: <dispatcher-tftp-ip>:<port>
```
If the value is not defined, LAVA will fallback to `dispatcher_ip`.
REST API
========
New TestJob endpoints
---------------------
New actions are available for TestJob REST API:
* ```/api/v0.2/jobs/${job_id}/resubmit/```
* ```/api/v0.2/jobs/${job_id}/cancel/```
* ```/api/v0.2/jobs/validate/```
Validate action expects a POST request with job definition data which is to
be validated against validation schema as argument.
New permissions endpoints
-------------------------
Additional new endpoints are available in REST API:
* ```/api/v0.2/permissions/devicetypes/```
* ```/api/v0.2/permissions/devices/```
They are used to control the new authorization system. Available only for
system administrators.
Bugs
----
Issue when non authenticated users where trying to list devices or device
types
which resulted in an error is now fixed.
Web server
==========
Gunicorn is now responsible for serving static files (css, js, ...). This
was handled by Apache2 before. This change is improving performances by
improving browser caching and pre-compressing resources.
This will allow to use any reverse proxy like HAProxy, nginx or Traefik.
Development
===========
Dispatcher remote debugging
---------------------------
Starting lava-slave with `--debug` will make it call `lava-run` with
`--debug`. And that will make the lava-run stop right before running each
job and start a remote pdb debugging session at that point.
The debugger is accessed via telnet.
Thanks
--
Rémi Duraffort
LAVA Architect
Linaro
Hi folks,
The 2020.01 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=================
Upgrading
=========
This release add database migrations to add one field to the workers
objects.
This migrations should be really quick due to the fairly small amount of
workers. However this mean that some downtime will be required.
Device-types
============
New device-types
----------------
New supported devices:
* sun50i-h6-pine-h64-model-b (Olimex)
docker
------
The docker DUTs can now use a specific network by setting `docker_networks`
in the device dictionary:
```jinja
{% set docker_networks = ["mynet", "mysecondnet"] %}
```
cmsis-dap
---------
cmsis-dap capable devices can now set `usb_sleep`.
This parameter will force lava to wait some seconds after the usb device
becomes visible.
sun50i devices
--------------
The initrd memory limit has been increased to fix a failure in u-boot with
large ramdisk. The new limit is set to `0xffffffffffffffff`.
REST API
========
We are expanding the REST API 0.2 functionalities in this release. New
methods
and endpoints are available for different models.
New endpoints with basic CRUD methods are available:
* /api/v0.2/tags/
* /api/v0.2/aliases/
Following endpoints have their functionalities expended so they now support
create, update and delete actions:
* /api/v0.2/workers/
* /api/v0.2/devices/
* /api/v0.2/devicetypes/
Devices endpoint now supports get and set dictionary via GET/POST request.
* /api/v0.2/devices/${hostname}/dictionary/
Similarly, devicetypes endpoint now supports retrieving and
creating/updating
the health checks and device type templates via GET/POST.
* /api/v0.2/devicetypes/${name}/health_check/
* /api/v0.2/devicetypes/${name}/template/
Workers endpoint now supports retrieving and creating/updating the
configuration and environment via GET/POST requests available here:
* /api/v0.2/workers/${hostname}/env/
* /api/v0.2/workers/${hostname}/config/
lava-dispatcher-host
====================
This release includes some new features towards being able to run
lava-dispatcher, or some test job actions, inside containers.
One of these is lava-dispatcher-host, a small package that provides the
minimal functionality that needs to run on the host OS: watching for udev
events for USB devices showing up, and making them available inside
containers.
For now, this only works for lxc containers, but it is separate from
lava-dispatcher itself so that when lava-dispatcher runs inside a docker
container, the amount of software needed on the host is minimal.
docker fastboot support
========================
This release also adds the possibility of deploying images and boot via a
fastboot binary that is running inside a docker container.
It simplifies AOSP deployment, which no longer need to instantiate a full
lxc container, install the tools inside of it, and boot it. The same can
now be achieved by using a pre-built docker images with the tools already
installed. For example, an AOSP deploy becomes:
```yaml
actions:
- deploy:
to: fastboot
docker:
image: my-fastboot-image
images:
ptable:
url: http://.../ptable.img
reboot: hard-reset
boot:
url: http://.../boot.img
reboot: hard-reset
# [... other images ...]
timeout:
minutes: 15
- boot:
method: fastboot
docker:
image: my-fastboot-image
prompts:
- 'healthd: No battery devices found'
- 'hikey: '
- 'console:'
timeout:
minutes: 15
```
Archive
=======
Some system images are compressed as a tarball (``.tar.*``), these images
need the ``archive`` option specified to unpack the system image correctly.
```yaml
- deploy:
images:
boot:
url: http://example.com/boot.tar.xz
compression: xz
archive: tar
```
The support for deploying qemu jobs using archive resources has been fixed.
Docker
======
Building images
---------------
The docker images are now built from sources. You can build the docker
image locally using:
```shell
.gitlab-ci/build/docker.sh dispatcher
.gitlab-ci/build/docker.sh server
```
lava-coordinator
----------------
`lava-coordinator` is now installed and started in `lava-server` docker
images.
lava-run
========
In previous releases, lava-run was executing host command under nice but
was not running itself under nice.
`lava-run` is now running under `nice` and the host command will not be
prepended anymore by `nice` as the niceness is inherited from the parent
process.
Worker concurrent limit
=======================
Admins can now limit the number of jobs running in parallel on the same
worker.
This limit can be set either from the admin interface or using lavacli:
```shell
lavacli workers update --job-limit 3 <worker>
```
Install script
==============
The `setup.py` script has been fully rewritten. This script is now used to
generate the Debian packages and the docker containers.
This script could be used to generate any distribution packages.
This script is now used to build docker containers from sources.
Settings
========
The django settings are stored in `/etc/lava-server/settings.conf` using
the json format.
Prior to this release, LAVA would silently ignore the settings if the file
was invalid. Starting from this release, LAVA services will refuse to start
if the file is not valid.
This will make errors easier to detect.
Thanks
--
Rémi Duraffort
LAVA Architect
Linaro
Hi folks,
The 2019.12 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=================
Device-types
============
New device-types
----------------
New supported devices:
* odroid-n2 (Amlogic)
* sun50i-h6-orangepi-3 (Olimex)
docker
------
A docker device can now use capabilities by setting `docker_capabilities`
in the device dictionary:
```jinja
{% set docker_capabilities = ["NET_ADMIN"] %}
```
TC2
---
Set `test_character_delay` to `50ms` by default to mitigate some serial
corruptions.
u-boot
------
In u-boot boot action, the `type` parameter has been removed. The type of a
kernel should be specified in the deploy action.
```yaml
- boot:
method: u-boot
type: bootz
```
Should be replaced by:
```yaml
- deploy:
to: tftp
kernel:
url: http://example.com/kernel
type: zimage
```
This support has been deprecated for more than a year.
Interactive test
================
echo: discard
-------------
When using interactive test with shells, LAVA will look for successes and
failures in the full output, including the command that was echoed by the
shell.
For example, when interacting with u-boot, LAVA will match the full output
of the DUT, including the shell echo:
```shell
=> echo "foo"
bar
```
LAVA would match the string `foo` echoed by the DUT.
This release adds a `echo` option allowing to skip the echoed string.
```yaml
- test:
interactive:
- name: network
prompts: ["=>", "/ # "]
echo: discard
script:
- name: dhcp
command: dhcp
[...]
```
unnamed commands
----------------
When using an interactive test action, if a command fail LAVA will:
* if the command is named: record the test case and execute the next command
* if the command is unnamed: raise a `TestError`
More information in the [documentation](
https://docs.lavasoftware.org/lava/actions-test.html#interactive).
Multinode
=========
Multinode jobs can now make use of `interactive` and `monitor` test
actions. However, the support for multinode is limited as these test
actions does not support synchronization.
If synchronization is required, use the `lava-test-shell` action.
REST API
========
Using display names when querying REST API
------------------------------------------
The choice fields in LAVA where previously queried in the following manner
in REST API when using filters:
```/api/v0.1/jobs/?health=2```
This is now fixed, fields like health and status, will use their display
names for filtering like so:
```/api/v0.1/jobs/?health=Incomplete```
API version bump
----------------
The new REST API version 0.2 is now available. New endpoints are available
for accessing the LAVA results (specifically test suites and test cases).
The old version is also available for compatibility with current API
clients.
New endpoints available:
* /api/v0.2/jobs/<job_id>/csv/
* /api/v0.2/jobs/<job_id>/yaml/
* /api/v0.2/jobs/<job_id>/suites/<suite_id>/
* /api/v0.2/jobs/<job_id>/suites/<suite_id>/csv/
* /api/v0.2/jobs/<job_id>/suites/<suite_id>/yaml/
* /api/v0.2/jobs/<job_id>/suites/<suite_id>/tests/
* /api/v0.2/jobs/<job_id>/suites/<suite_id>/tests/<test_id>/
Old endpoints removed:
* /api/v0.2/jobs/<job_id>/tests/
lxc protocol
============
Jobs using the lxc protocol to update resources inside the lxc test shell,
can now have access to the `NFS_ROOTFS` and `NFS_SERVER_IP` environment
variable.
This will allow jobs to boot using adb and nfs.
Database migrations
===================
This release includes a database migration to remove effects of an old bug
that was creating duplicated database objects. The duplicated objects are
anyway not accessible from the web interface nor the APIs.
The duplicated `TestData` objects will be removed from the database.
Device config schema
====================
`lava-schema`, the tool currently used to check job definition schema, can
now also check for device configuration schema.
```shell
/usr/share/lava-common/lava-schema.py device device.jinja2
```
By default, `lava-schema` will expect a `jinja2` template as first
argument. This file would be rendered prior to run the schema checker.
To check an already rendered device dictionary, add `--no-render`.
```shell
/usr/share/lava-common/lava-schema.py device --no-render device.yaml
```
By default, `lava-schema` will look for base templates in
`/etc/lava-server/dispatcher-config/device-types`. You can specify the path
to use by adding `--path``.
Checksum
========
LAVA is now able to check `sha512sum` checksums of downloaded resources:
```yaml
- deploy:
images:
system:
url: http://example.com/system.img.xz
compression: xz
sha512sum:
e0e82b5adfae84ff97f4f6488e5b4c64b0dfc7ad8a37b4bcbb887d9f85a6be0a
```
Test shell variables
=====================
Users can now set environment variables in the lava test shell by adding
`environment` to the job definition:
```yaml
environment:
FOO: bar
BAR: baz
```
The variables will be made available in the lava test shell.
More information in the [documentation](
https://docs.lavasoftware.org/lava/dispatcher-format.html#provide-environme…
).
Performances
============
The performances of the lava-server services has been dramatically improved.
With this release, the lava server and master can schedule, start and
handle 2.5 times more parallel jobs than in the previous version.
Moreover submitting jobs is now 3 times faster.
We will continue in the following releases to improve the performances.
Caching
=======
LAVA dispatcher can now make use of simple caching service like [KissCache](
https://git.lavasoftware.org/ivoire/KissCache).
Admin would have to update the dispatcher configuration in
`/etc/lava-server/dispatcher.d/<hostname>/dispatcher.yaml`:
```shell
# Set this variable when using http caching service based on url
substitution
# like KissCache
# When downloading resources, lava dispatcher will use this formating string
# instead of the original url.
http_url_format_string: "https://cache.lavasoftware.org/api/v1/fetch/?url=%s
"
```
More information in the [documentation](
https://docs.lavasoftware.org/lava/proxy.html#using-the-http-cache).
Bazaar
======
The support for pulling test definition from bazaar repositories has been
dropped due to the lack of users.
Thanks
--
Rémi Duraffort
LAVA Architect
Linaro
Hello,
following the release of LAVA 2019.11, we released lavacli v0.9.9
This release does:
* fix a crash with LAVA >= 2019.09
* improve log colors
* allow to follow logs when submitting jobs
As usual, lavacli could be installed from:
* pypi (https://pypi.org/project/lavacli/)
* debian (https://tracker.debian.org/pkg/lavacli) (coming soon).
Cheers
--
Rémi Duraffort
LAVA Architect
Linaro
Hi folks,
The 2019.11 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=================
Device-types
============
New device-types
----------------
New supported devices:
* imx8mn-ddr4-evk
* meson-sm1-sei610 (Amlogic)
* r8a7796-m3ulcb-kf
udev events
===========
Fix a race condition when waiting for `udev events`. If the dispatcher is
under load, sometimes the udev event will be sent before `lava-run` start
waiting for event.
In such case, the event is lost and LAVA will wait forever.
LAVA is now waiting for the event and at the same time looking at the usb
devices. If the event is missed, LAVA will notice the presence of the usb
device.
Documentation
=============
The documentation has been improved. The job definition snippets are now
easier to understand and to copy&paste into your own job definition.
This is an ongoing work that will continue in the following releases.
Job definition schema
=====================
Job context
-----------
The schema validator is now checking the content of the `context`
dictionary for both single and multinode jobs.
A new set of keys has been added: `custom_kernel_args` and `no_kvm`
Only the following keys are now allowed:
* `arch`, `boot_console`, `boot_root`, `cpu`, `extra_options`,
`guestfs_driveid`, `guestfs_interface`, `guestfs_size`, `machine`,
`memory`, `model`, `monitor`, `netdevice`, `no_kvm`, `serial`, `vga`
* `booti_dtb_addr`, `booti_kernel_addr`, `booti_ramdisk_addr`,
`bootm_dtb_addr`, `bootm_kernel_addr`, `bootm_ramdisk_addr`,
`bootz_dtb_addr`, `bootz_kernel_addr`, `bootz_ramdisk_addr`
* `boot_character_delay`, `bootloader_prompt`, `console_device`,
`custom_kernel_args`, `extra_kernel_args`, `extra_nfsroot_args`,
`kernel_loglevel`, `kernel_start_message`, `lava_test_results_dir`,
`menu_interrupt_prompt`, `mustang_menu_list`, `test_character_delay`,
`tftp_mac_address`, `uboot_extra_error_message`, `uboot_needs_interrupt`
Jobs using keys that are not listed in this list will be rejected.
If you think that some more keys should be whitelisted, talk to us.
Debian packaging
================
The `lava-dispatcher` Debian package architecture is now set to `All` as
the code is architecture independent.
Administrators
==============
Device dictionary
-----------------
Admins can now check device configuration using the `lava-server` command
line:
```shell
lava-server manage devices check qemu-01
lava-server manage devices check --all
```
Controlling devices
------------------
Admins can now use the control commands specified into the device
dictionary manually:
```shell
lava-server manage devices control <hostname> <command>
```
The commands are: `on`, `off`, `reset` and `connect`
Docker
======
The `lava-server` and `lava-dispatcher` docker images are now based on
debian `buster`.
Thanks
--
Rémi Duraffort
LAVA Architect
Linaro
Hi folks,
The 2019.10 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and will be published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=================
Device-types
============
New device-types
----------------
New supported devices:
* hifive-unleashed-a00 (SiFi)
* meson-g12b-a311d-khadas-vim3 (Amlogic)
* meson-gxm-q200 (Amlogic)
* MIMXRT1050-EVK (NXP)
* sun4i-a10-olinuxino-lime (Olimex)
* sun7i-a20-olinuxino-lime2 (Olimex)
* sun7i-a20-olinuxino-micro (Olimex)
u-boot error messages
---------------------
The list of u-boot errors that LAVA can recognized has been extended. The
full list is now: `Resetting CPU`, `Must RESET board to recover`,
`TIMEOUT`, `Retry count exceeded`, `Retry time exceeded; starting again`,
`ERROR: The remote end did not respond in time.`, `Bad Linux ARM64 Image
magic!`, `Wrong Ramdisk Image Format`, `Ramdisk image is corrupt or
invalid`, `ERROR: Failed to allocate`, `TFTP error: trying to overwrite
reserved memory`
dragonboard
-----------
Update the device-type template to flash GPT partitions in the right order
(if present).
meson-g12a-sei510
-----------------
Fix some issues with u-boot command corruptions and allow to use LAVA tftp
resources already setup by LAVA.
imx6q-sabrelite
---------------
Move the addresses around for imx6q-sabrelite to give `63MiB` for the
kernel image, `1MiB` for the dtb and the rest for the ramdisk.
jlink
-----
LAVA can now boot some boards using the `jlink` boot method. Currently,
only the `frdm-k64f` can use this method.
```yaml
- deploy:
timeout:
minutes: 3
to: tmpfs
images:
zephyr:
url: http://zephyr.bin
- boot:
method: jlink
timeout:
minutes: 10
```
vemsd
-----
The `vemsd` support has been improved after some experiments in the
Cambridge lab:
1. call `sync` on the mount point prior to umount
2. raise the right exception when failing to flash (Infrastructure error)
mps
---
The `mps` support was also improved after some issues in the Cambridge lab:
1. allow to flash multiple test binaries in one deploy block
2. allow to use soft reboot
```yaml
- deploy:
to: mps
images:
recovery_image:
url: mps2_sse200_an512.tar.gz
compression: gz
test_binary_1:
url: tfm_sign.bin
test_binary_2:
url: mcuboot.bin
```
Authorization
=============
Some issues found with the new authorization model has been fixed:
* fix crashes in some XMLRPC calls
* merge `admin_testjob` and `cancel_resubmit_testjob` into `change_testjob`
* remove `submit_testjob` and `add_testjob` permissions
* remove old permissions like `dashoard_app`
* rename `admin_` permissions to `change_` permissions
The [documentation](https://docs.lavasoftware.org/lava/authorization.html)
of this new model has been updated.
gunicorn
========
Fix log rotation. Due to a missing reload in the log rotate configuration
file, the `lava-server-gunicorn` service was logging to the old log file,
even after the log rotation.
Admin should look at the logs in `/var/log/lava-server/`.
Docker
======
Add an option to `lava-master` to set the event url when `lava-master` and
`lava-publisher` are running on two different hosts or containers.
Add `EVENT_URL="--event-url tcp://localhost:5500"` in
`/etc/lava-server/lava-master`.
This setting is used by [docker-compose](
https://git.lavasoftware.org/lava/pkg/docker-compose/).
Rest API
========
Users can now submit jobs by `POST` the job definition to the `/jobs/`
endpoint.
Django 2
========
This release is the first release to work on both Django 1.11 and Django 2.
Development
===========
Allow developer to run lava server and dispatcher services using `foremon`.
Thanks to this support, running a development version of lava is now way
simpler.
--
Rémi Duraffort
LAVA Architect
Linaro
Hi folks,
The 2019.07 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=======================
Device-types
============
New device-types
----------------
New supported devices:
* meson-gxl-s805x-libretech-ac (Amlogic)
* meson-gxl-s905d-p230 (Amlogic)
* meson-gxl-s905x-p212 (Amlogic)
* meson8b-ec100 (Amlogic)
* sun50i-h6-pine-h64
ox820-cloudengines-pogoplug-series-3
------------------------------------
Some ARM defconfig changes made the kernel bigger and the ox820 device
cannot boot any more. Update the ramdisk address to **0x61000000**
(was **0x60e00000**).
extra_nfsroot_args
------------------
Some device-types (`jetson-tk1`, `juno`, `qcom-qdf2400`, `rzn1d`,
`synquacer` and `tc2`) were not allowing the job submitter to set
`extra_nfsroot_args` in the job context.
Job submitters can now pass extra nfs rootfs arguments in the job
context for these devices.
context:
extra_nfsroot_args: ",wsize=65536 nfsrootdebug"
Database
========
Migrations
----------
This new version creates several database migrations:
* add `ExtendedUser.table_length` field
* add new tables: `GroupDeviceTypePermission` and `GroupDevicePermission`
* add custom permissions for Device: `view_device`, `submit_to_device`,
`admin_device`
* add custom permissions for DeviceType: `view_devicetype`,
`submit_to_devicetype`, `admin_devicetype`
* remove fields from DeviceType: `owners_only`
* remove fields from Device: `user`, `group`, `is_public`
* remove fields from TestJob: `user`, `group`, `visibility`
* delete table: `DefaultDeviceOwner`
These may incur a small amount of downtime during the upgrade process
as database migrations happen. This may take a long time if there is a
large number of records in the ``TestJob`` table.
A goal for the LAVA development team is to reduce the frequency of
database migrations to reduce this cost as much as possible. This
release also includes a data migration and adding new records in the
newly created tables so that authorization refactoring does not change
any existing user-visible behaviour.
Authorization refactoring
=========================
Due to scaling problems with the previous authentication model, the
way LAVA controls access to its most important entities (DeviceType,
Device and TestJob) has been completely revamped.
Previously, LAVA used flags like `owners_only`, `user`, `group`,
`visibility`. On large LAVA installations these were causing
significant database overhead. They have now been removed and replaced
by a new design. Two new tables have been added:
`GroupDeviceTypePermission` and `GroupDevicePermission`. The
permission model has been moved to Group based control and
administrators can control those through the Django admin interfaces
for ``DeviceType`` and ``Device`` records, or via the respective
XMLRPC calls. The data migration will convert records across from the
old auth model to the new model, keeping equivalent permissions.
The new permission model is inheritable by design so any permissions
applied to device types will also be applied to devices belonging to
the particular device type, and all permissions applied to a device
will be relevant for permissions on test jobs running on that
particular device. Further information can be found in the LAVA
documentation.
Table size
==========
The LAVA web interface displays a lot of tables. The size of the table
is configurable at instance level by the admin since LAVA
[2019.06](2019.06). There is a default table length defined in the
system.
In this release, users can now also configure the default table length
to use when logged in. This can be set by the user in their profile
page at `https://<host>/me/`.
Steve, for the LAVA team.
--
Steve McIntyre steve.mcintyre(a)linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
