Hi folks,
The 2019.11 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=================
Device-types
============
New device-types
----------------
New supported devices:
* imx8mn-ddr4-evk
* meson-sm1-sei610 (Amlogic)
* r8a7796-m3ulcb-kf
udev events
===========
Fix a race condition when waiting for `udev events`. If the dispatcher is
under load, sometimes the udev event will be sent before `lava-run` start
waiting for event.
In such case, the event is lost and LAVA will wait forever.
LAVA is now waiting for the event and at the same time looking at the usb
devices. If the event is missed, LAVA will notice the presence of the usb
device.
Documentation
=============
The documentation has been improved. The job definition snippets are now
easier to understand and to copy&paste into your own job definition.
This is an ongoing work that will continue in the following releases.
Job definition schema
=====================
Job context
-----------
The schema validator is now checking the content of the `context`
dictionary for both single and multinode jobs.
A new set of keys has been added: `custom_kernel_args` and `no_kvm`
Only the following keys are now allowed:
* `arch`, `boot_console`, `boot_root`, `cpu`, `extra_options`,
`guestfs_driveid`, `guestfs_interface`, `guestfs_size`, `machine`,
`memory`, `model`, `monitor`, `netdevice`, `no_kvm`, `serial`, `vga`
* `booti_dtb_addr`, `booti_kernel_addr`, `booti_ramdisk_addr`,
`bootm_dtb_addr`, `bootm_kernel_addr`, `bootm_ramdisk_addr`,
`bootz_dtb_addr`, `bootz_kernel_addr`, `bootz_ramdisk_addr`
* `boot_character_delay`, `bootloader_prompt`, `console_device`,
`custom_kernel_args`, `extra_kernel_args`, `extra_nfsroot_args`,
`kernel_loglevel`, `kernel_start_message`, `lava_test_results_dir`,
`menu_interrupt_prompt`, `mustang_menu_list`, `test_character_delay`,
`tftp_mac_address`, `uboot_extra_error_message`, `uboot_needs_interrupt`
Jobs using keys that are not listed in this list will be rejected.
If you think that some more keys should be whitelisted, talk to us.
Debian packaging
================
The `lava-dispatcher` Debian package architecture is now set to `All` as
the code is architecture independent.
Administrators
==============
Device dictionary
-----------------
Admins can now check device configuration using the `lava-server` command
line:
```shell
lava-server manage devices check qemu-01
lava-server manage devices check --all
```
Controlling devices
------------------
Admins can now use the control commands specified into the device
dictionary manually:
```shell
lava-server manage devices control <hostname> <command>
```
The commands are: `on`, `off`, `reset` and `connect`
Docker
======
The `lava-server` and `lava-dispatcher` docker images are now based on
debian `buster`.
Thanks
--
Rémi Duraffort
LAVA Architect
Linaro
Hi folks,
The 2019.10 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and will be published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=================
Device-types
============
New device-types
----------------
New supported devices:
* hifive-unleashed-a00 (SiFi)
* meson-g12b-a311d-khadas-vim3 (Amlogic)
* meson-gxm-q200 (Amlogic)
* MIMXRT1050-EVK (NXP)
* sun4i-a10-olinuxino-lime (Olimex)
* sun7i-a20-olinuxino-lime2 (Olimex)
* sun7i-a20-olinuxino-micro (Olimex)
u-boot error messages
---------------------
The list of u-boot errors that LAVA can recognized has been extended. The
full list is now: `Resetting CPU`, `Must RESET board to recover`,
`TIMEOUT`, `Retry count exceeded`, `Retry time exceeded; starting again`,
`ERROR: The remote end did not respond in time.`, `Bad Linux ARM64 Image
magic!`, `Wrong Ramdisk Image Format`, `Ramdisk image is corrupt or
invalid`, `ERROR: Failed to allocate`, `TFTP error: trying to overwrite
reserved memory`
dragonboard
-----------
Update the device-type template to flash GPT partitions in the right order
(if present).
meson-g12a-sei510
-----------------
Fix some issues with u-boot command corruptions and allow to use LAVA tftp
resources already setup by LAVA.
imx6q-sabrelite
---------------
Move the addresses around for imx6q-sabrelite to give `63MiB` for the
kernel image, `1MiB` for the dtb and the rest for the ramdisk.
jlink
-----
LAVA can now boot some boards using the `jlink` boot method. Currently,
only the `frdm-k64f` can use this method.
```yaml
- deploy:
timeout:
minutes: 3
to: tmpfs
images:
zephyr:
url: http://zephyr.bin
- boot:
method: jlink
timeout:
minutes: 10
```
vemsd
-----
The `vemsd` support has been improved after some experiments in the
Cambridge lab:
1. call `sync` on the mount point prior to umount
2. raise the right exception when failing to flash (Infrastructure error)
mps
---
The `mps` support was also improved after some issues in the Cambridge lab:
1. allow to flash multiple test binaries in one deploy block
2. allow to use soft reboot
```yaml
- deploy:
to: mps
images:
recovery_image:
url: mps2_sse200_an512.tar.gz
compression: gz
test_binary_1:
url: tfm_sign.bin
test_binary_2:
url: mcuboot.bin
```
Authorization
=============
Some issues found with the new authorization model has been fixed:
* fix crashes in some XMLRPC calls
* merge `admin_testjob` and `cancel_resubmit_testjob` into `change_testjob`
* remove `submit_testjob` and `add_testjob` permissions
* remove old permissions like `dashoard_app`
* rename `admin_` permissions to `change_` permissions
The [documentation](https://docs.lavasoftware.org/lava/authorization.html)
of this new model has been updated.
gunicorn
========
Fix log rotation. Due to a missing reload in the log rotate configuration
file, the `lava-server-gunicorn` service was logging to the old log file,
even after the log rotation.
Admin should look at the logs in `/var/log/lava-server/`.
Docker
======
Add an option to `lava-master` to set the event url when `lava-master` and
`lava-publisher` are running on two different hosts or containers.
Add `EVENT_URL="--event-url tcp://localhost:5500"` in
`/etc/lava-server/lava-master`.
This setting is used by [docker-compose](
https://git.lavasoftware.org/lava/pkg/docker-compose/).
Rest API
========
Users can now submit jobs by `POST` the job definition to the `/jobs/`
endpoint.
Django 2
========
This release is the first release to work on both Django 1.11 and Django 2.
Development
===========
Allow developer to run lava server and dispatcher services using `foremon`.
Thanks to this support, running a development version of lava is now way
simpler.
--
Rémi Duraffort
LAVA Architect
Linaro
Hi folks,
The 2019.07 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=======================
Device-types
============
New device-types
----------------
New supported devices:
* meson-gxl-s805x-libretech-ac (Amlogic)
* meson-gxl-s905d-p230 (Amlogic)
* meson-gxl-s905x-p212 (Amlogic)
* meson8b-ec100 (Amlogic)
* sun50i-h6-pine-h64
ox820-cloudengines-pogoplug-series-3
------------------------------------
Some ARM defconfig changes made the kernel bigger and the ox820 device
cannot boot any more. Update the ramdisk address to **0x61000000**
(was **0x60e00000**).
extra_nfsroot_args
------------------
Some device-types (`jetson-tk1`, `juno`, `qcom-qdf2400`, `rzn1d`,
`synquacer` and `tc2`) were not allowing the job submitter to set
`extra_nfsroot_args` in the job context.
Job submitters can now pass extra nfs rootfs arguments in the job
context for these devices.
context:
extra_nfsroot_args: ",wsize=65536 nfsrootdebug"
Database
========
Migrations
----------
This new version creates several database migrations:
* add `ExtendedUser.table_length` field
* add new tables: `GroupDeviceTypePermission` and `GroupDevicePermission`
* add custom permissions for Device: `view_device`, `submit_to_device`,
`admin_device`
* add custom permissions for DeviceType: `view_devicetype`,
`submit_to_devicetype`, `admin_devicetype`
* remove fields from DeviceType: `owners_only`
* remove fields from Device: `user`, `group`, `is_public`
* remove fields from TestJob: `user`, `group`, `visibility`
* delete table: `DefaultDeviceOwner`
These may incur a small amount of downtime during the upgrade process
as database migrations happen. This may take a long time if there is a
large number of records in the ``TestJob`` table.
A goal for the LAVA development team is to reduce the frequency of
database migrations to reduce this cost as much as possible. This
release also includes a data migration and adding new records in the
newly created tables so that authorization refactoring does not change
any existing user-visible behaviour.
Authorization refactoring
=========================
Due to scaling problems with the previous authentication model, the
way LAVA controls access to its most important entities (DeviceType,
Device and TestJob) has been completely revamped.
Previously, LAVA used flags like `owners_only`, `user`, `group`,
`visibility`. On large LAVA installations these were causing
significant database overhead. They have now been removed and replaced
by a new design. Two new tables have been added:
`GroupDeviceTypePermission` and `GroupDevicePermission`. The
permission model has been moved to Group based control and
administrators can control those through the Django admin interfaces
for ``DeviceType`` and ``Device`` records, or via the respective
XMLRPC calls. The data migration will convert records across from the
old auth model to the new model, keeping equivalent permissions.
The new permission model is inheritable by design so any permissions
applied to device types will also be applied to devices belonging to
the particular device type, and all permissions applied to a device
will be relevant for permissions on test jobs running on that
particular device. Further information can be found in the LAVA
documentation.
Table size
==========
The LAVA web interface displays a lot of tables. The size of the table
is configurable at instance level by the admin since LAVA
[2019.06](2019.06). There is a default table length defined in the
system.
In this release, users can now also configure the default table length
to use when logged in. This can be set by the user in their profile
page at `https://<host>/me/`.
Steve, for the LAVA team.
--
Steve McIntyre steve.mcintyre(a)linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
Hi folks,
The 2019.07 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and will be available at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=======================
Device-types
============
New device-types
----------------
New supported devices:
* sun50i-h6-orangepi-one-plus (Allwinner)
* sun8i-a33-olinuxino (Olimex)
* TI CC13X2 LaunchPad (TI)
cmsis-dap
---------
In order to fix some infrequent issues on specific hardware and boards,
LAVA will now explicitly perform a sync on the directory before umounting
it.
depthcharge
-----------
### extra kernel arguments
Admins and users can now both define some extra kernel arguments in
device-type templates and in the job context.
Use `base_kernel_args` in device-type templates and `extra_kernel_args` in
job definitions.
### LZMA compression
Enable LZMA compression for arm64 kernel Image with Depthcharge.
docker
------
Secondary ssh connection is now supported for the docker device-type. Look
at the [documentation](
https://docs.lavasoftware.org/lava/pipeline-writer-secondary.html#secure-sh…)
for more information.
lxc
---
Under some circumstances, LAVA could leak the lxc rootfs in
`/var/lib/lxc/`. LAVA will no longer leak such rootfs.
We advice admins to have a look at `/var/lib/lxc` and to remove unused
rootfs as they can use a significant amount of disk space.
OpenOCD
-------
Admins can set `board_selection_cmd` in the device-type template to specify
a command to pass the board id/serial number to OpenOCD.
See OpenOCD documentation for details on the command to set the serial
number for the interface your device type is using.
Docker
======
Dockerfiles
-----------
The repository [lava/pkg/docker](
https://git.lavasoftware.org/lava/pkg/docker), hosting the Dockerfiles has
been merged into the [lava/lava](https://git.lavasoftware.org/lava/lava)
repository.
This merge will allow (in a near future) to build the docker images from
sources.
entrypoint.d
------------
When starting `lava-server` or `lava-dispatcher` docker images, admin can
execute custom scripts by adding them into `/root/entrypoint.d/` The
scripts will be executed right before the start of the services.
In this release the support for custom script has been fixed.
performances
------------
When running gunicorn in docker container, the service will now use
`/dev/shm` for storing heartbeat artifacts.
By default, gunicorn will use /tmp which is not mounted in docker
containers. `/dev/shm` is guaranteed to be available and an in-memory
filesystem.
Using a non-memory file system can lead to intermittent hang.
See [gunicorn in docker](
https://pythonspeed.com/articles/gunicorn-in-docker/) for more information.
permissions
-----------
When starting `lava-logs` or `lava-server-gunicorn` the entrypoint will now
check that the files under `/etc/lava-server/dispatcher-config/` and
`/var/lib/lava-server/default/media/` belong to the `lavaserver` user and
group.
The docker container will refuse to start if that's not the case as this
can lead to crashes or strange behaviors.
lava-publisher
==============
The lava-publisher sockets will now automatically send ping messages thanks
to the [ZMTP protocol](https://rfc.zeromq.org/spec:37/ZMTP/).
If the connection is lost, the [heartbeating protocol](
https://rfc.zeromq.org/spec:37/ZMTP/#connection-heartbeating) will detect
the issue and try to reconnect.
Error reporting
===============
In order to track crashes of the LAVA services, admins can setup [sentry](
https://sentry.io) in `/etc/lava-server/settings.conf`:
```json
"SENTRY_DSN": "http://<public_key>@<sentry>/<project-id>"
```
You should also install `sentry-sdk` from [pypi](
https://pypi.org/project/sentry-sdk/) with:
```shell
apt-get install python3-pip
python3 -m pip install sentry-sdk==0.10.2
```
The lava-server docker image comes with the sentry sdk already installed.
Rémi, for the LAVA team.
--
Rémi Duraffort
LAVA Team, Linaro
Hello,
following the release of LAVA 2019.06, we released lavacli v0.9.7
This release brings only one new feature: get/set worker environment
In order to use this feature, you need LAVA >= 2019.06
Then you can:
lavacli workers env get <hostname>
lavacli workers env set <hostname> <env.yaml>
As usual, lavacli could be installed from:
* pypi (https://pypi.org/project/lavacli/)
* debian (https://tracker.debian.org/pkg/lavacli) (coming soon).
Cheers
--
Rémi Duraffort
LAVA Team, Linaro
Hi folks,
The 2019.06 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are available at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=======================
Device-types
============
New device-types
----------------
New supported devices:
* dragonboard-845c
* meson-g12b-odroid-n2 (Amlogic)
* ox820-cloudengines-pogoplug-series-3 (CloudEngines)
* stm32mp157c-dk2 (STMicroelectronics)
* sun50i-a64-bananapi-m64 (Allwinner)
* sun7i-a20-cubieboard2 (Allwinner)
* sun8i-a83t-bananapi-m3 (Allwinner)
* sun8i-r40-bananapi-m2-ultra (Allwinner)
docker
------
Add an option to start docker device as privileged. Add `{% set
docker_privileged = True %}` to the device dictionary.
qemu
----
Add support for `nios2` and `xtensa` architectures.
ssh
---
The ssh username can now be set in the device dictionary.
Boot methods
============
OpenOCD
-------
A new boot method called `openocd` has been contributed. This method,
allows to flash and boot boards using `openocd`.
An openocd job would look like:
- deploy:
to: tmpfs
images:
binary:
url: http://.../zephyr.elf
openocd_script:
url: http://.../cc3220_xds110.cfg
- boot:
method: openocd
- test:
monitors:
- name: tests
start: (tc_start\(\)|starting .*test|BOOTING ZEPHYR OS)
end: PROJECT EXECUTION SUCCESSFUL
pattern: (?P<result>(PASS|FAIL))\s-\s(?P<test_case_id>\w+)\r\n
fixupdict:
PASS: pass
FAIL: fail
For the moment, only the `cc3220SF` board supports this method.
Test shell
==========
result strings
---------------
In LAVA versions prior to 2019.01 both lower and upper case results
were accepted. This behavior was changed in 2019.01 to only accept
lower case.
In order to keep compatibility among test jobs, the upper case result
strings (`PASS`, `FAIL`, `SKIP` and `UNKNOWN`) are now valid again.
environment
-----------
A new variable, called `environment` has been added to the device
dictionaries. These can be used, for example, to describe physical
hardware connections between the DUT and interfaces on the worker or
other addressable hardware.
For ease of use, LAVA will directly export the content of the
`device_info`, `environment`, `static_info` and `storage_info`
dictionaries into the test shell environment. The dictionaries and
lists will be unrolled, for example:
{% set environment = {"RELAY_ADDRESS": "10.66.16.103", "REMOTE_SERIAL_PORT": "/dev/ttyUSB2"} %}
{% set static_info = [{"board_id": "S_NO81730000"}, {"board_id": "S_NO81730001"}] %}
{% set storage_info = [{'SATA': '/dev/disk/by-id/ata-ST500DM002-1BD142_W3T79GCW'}] %}
will become:
export RELAY_ADDRESS='10.66.16.103'
export REMOTE_SERIAL_PORT='/dev/ttyUSB2'
export LAVA_STATIC_INFO_0_board_id='S_NO81730000'
export LAVA_STATIC_INFO_1_board_id='S_NO81730001'
export LAVA_STORAGE_INFO_0_SATA='/dev/disk/by-id/ata-ST500DM002-1BD142_W3T79GCW'
REST API
========
Filtering has been activated in the REST API for 'devices',
'device-types' and 'jobs'. See the
[documentation](https://docs.lavasoftware.org/lava/data-export.html#filterin…
for the details.
Table size
==========
The LAVA web interface displays a lot of tables. The size of the table
is now configurable at instance level by the admin.
Update `settings.conf` and restart `lava-server-gunicorn`:
"DEFAULT_TABLE_LENGTH": 50,
By default, every table will display 25 elements.
lava-slave
==========
Socks proxy
-----------
When connecting with `lava-master` and `lava-logs`, `lava-slave` can
now use a socks proxy if needed.
In order to use a socks proxy, you should either:
* start lava-slave with `--socks-proxy localhost:8081`
* edit `/etc/lava-dispatcher/lava-slave` and add
`SOCKS_PROXY="--socks-proxy localhost:8081"`
The support is now complete.
Job definition schema
=====================
Job context
-----------
The schema validator is now checking the content of the `context`
dictionary for both single and multinode jobs.
A new set of keys has been added: `boot_character_delay`,
`booti_dtb_addr`, `booti_kernel_addr`, `booti_ramdisk_addr`,
`bootm_dtb_addr`, `bootm_kernel_addr`, `bootm_ramdisk_addr`,
`bootz_dtb_addr`, `bootz_kernel_addr`, `bootz_ramdisk_addr`.
Only the following keys are now allowed: * `arch`, `boot_console`,
`boot_root`, `cpu`, `extra_options`, `guestfs_driveid`,
`guestfs_interface`, `guestfs_size`, `machine`, `memory`, `model`,
`monitor`, `netdevice`, `serial`, `vga` * `booti_dtb_addr`,
`booti_kernel_addr`, `booti_ramdisk_addr`, `bootm_dtb_addr`,
`bootm_kernel_addr`, `bootm_ramdisk_addr`, `bootz_dtb_addr`,
`bootz_kernel_addr`, `bootz_ramdisk_addr` * `boot_character_delay`,
`bootloader_prompt`, `console_device`, `extra_kernel_args`,
`extra_nfsroot_args`, `kernel_loglevel`, `kernel_start_message`,
`lava_test_results_dir`, `menu_interrupt_prompt`, `mustang_menu_list`,
`test_character_delay`, `tftp_mac_address`,
`uboot_extra_error_message`, `uboot_needs_interrupt`
Jobs using keys that are not listed in this list will be rejected.
If you think that some more keys should be whitelisted, talk to us.
XML-RPC
=======
Two functions called `scheduler.workers.get_env` and
`scheduler.workers.set_env` have been introduced. These functions
allow users to get and set the environment for a given dispatcher.
The corresponding command line has been added to `lavacli` v0.9.7:
$ lavacli workers env get dispatcher01
$ lavacli workers env set dispatcher01 env.yaml
Steve, for the LAVA team
--
Steve McIntyre steve.mcintyre(a)linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
Hi folks,
We found a significant bug in our latest 2019.05 release, affecting
the download of large compressed log files. We have released a hotfix
update for this issue, tagged as 2019.05post1.
.deb packages have been built in GitLab CI and are available at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
will be available shortly from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=======================
Full list of changes since 2019.04
==================================
8b309eadb Pass iterator to StreamingHttpReponse
Steve, for the LAVA team
--
Steve McIntyre steve.mcintyre(a)linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
Hello,
following the release of LAVA 2019.05, we released lavacli v0.9.6
This release brings some improvements:
* "jobs cancel" can now cancel multiple jobs
* "utils templates render" accepts a context and can trim the output
* add a timeout to "jobs wait" command
In order to use the new aliases introduced/fixed in LAVA 2019.05, lavacli
v0.9.6 is required.
As usual, lavacli could be installed from:
* pypi (https://pypi.org/project/lavacli/)
* debian (https://tracker.debian.org/pkg/lavacli) (coming soon).
Cheers
--
Rémi Duraffort
LAVA Team, Linaro
Hi folks,
The 2019.05 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are available at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
will be available shortly from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=======================
Device-types
============
Aliases
-------
The device-type aliases feature has been totally redesigned. You can
now define multiple aliases for a given device-type. In a job
definition, users can use either the device-type or any of its
corresponding aliases.
New device-types
----------------
New supported devices:
* da850-lcdk (davinci)
* SoCA9 (Schneider)
frdm-k64f
---------
It is now possible to override the setting `resets_after_flash` in the
device dictionary, to support newer firmwares.
meson-gl12
----------
Fix the dtb and ramdisk addresses:
* dtb: **0x01070000** (was **0x0107f000**)
* ramdisk: **0x08000000** (was **0x04000000**)
RPi
---
Move the ramdisk load address to **0x02200000** (was **0x02100000**)
to allow more space for the kernel image and avoid them overlapping.
rzn1d
-----
Allow to use zImage.
Database
========
Migrations
----------
This new version creates several database migrations to:
* remove `TestJob.admin_notifications` field
* improve `DeviceType` aliases
* replace some CharField types with TextField to remove unnecessary
size limitations
These may incur a small amount of downtime during the upgrade process
as database migrations happen. A goal for the LAVA development team is
to reduce the frequency of database migrations to reduce this cost as
much as possible. The next **planned** set of database migrations is
in the 2019.08 release, in August.
dashboard_app
-------------
The dashboard_app (holding v1 results) has been fully removed,
including:
* migrations
* database model
* xmlrpc calls
Docker
======
multiarch
---------
Due to some limitations in the docker registry embedded in GitLab, we
are no longer building and providing multiarch docker images for every
commit. Multiarch images will still be available on [docker
hub](https://hub.docker.com/u/lavasoftware/) for every release.
lava-dispatcher
---------------
The dispatcher docker image now includes `docker-ce-cli`. This will
allow lava to control docker containers from within a lava-dispatcher
container, helping admins to participate in lavafed.
Server performances
===================
Template caching
----------------
The `USE_TEMPLATE_CACHE` setting variable has been removed. On newer
Django versions, this option is no longer useful and might even cause
problems.
Compressing job logs
--------------------
Support has been added for compressing log files on the server, using
xz (lzma) compression. Logs will be stored uncompressed as jobs are
running, but they may later be compressed by admins to reduce the
amount of space needed.
The `lava-server manage jobs compress` command
includes options to allow for management of this. Admins can use this
in cronjobs to automatically compress job logs older than 30 days, or
similar (`lava-server manage jobs compress --older-than 30d --slow`).
LAVA will automatically decompress old compressed logs on the fly if
they are ever needed again in future.
Job definition schema
=====================
Admin helper
------------
The `lava-server manage jobs validate` command can now send a mail to
the admin when an invalid job is found.
To make use of this, admins can setup a regular cronjob that will call:
```shell
lava-server manage jobs validate --mail-admins
```
Job context
-----------
The schema validator is now checking the content of the `context`
dictionary for both single and multinode jobs.
Two new allowed keys have been added: `uboot_extra_error_message` and
`uboot_needs_interrupt`
Only the following keys are now allowed:
* `arch`, `boot_console`, `boot_root`, `cpu`, `extra_options`, `guestfs_driveid`, `guestfs_interface`, `guestfs_size`, `machine`, `memory`, `model`, `monitor`, `netdevice`, `serial`, `vga`
* `bootloader_prompt`, `console_device`, `extra_kernel_args`, `extra_nfsroot_args`, `kernel_loglevel`, `kernel_start_message`, `lava_test_results_dir`, `menu_interrupt_prompt`, `mustang_menu_list`, `test_character_delay`, `tftp_mac_address`, `uboot_extra_error_message`, `uboot_needs_interrupt`
Jobs using keys that are not listed in this list will be rejected.
If you think that some more keys should be whitelisted, talk to us.
Extending job context
---------------------
Admins can extend the white list by updating `EXTRA_CONTEXT_VARIABLES`
in the settings:
```json
"EXTRA_CONTEXT_VARIABLES": ["custom_variable1", "variable_2"]
```
Don't forget to restart `lava-server-gunicorn` if you make changes
here.
lava-slave
==========
Socks proxy
-----------
When connecting with `lava-master` and `lava-logs`, `lava-slave` can
now use a socks proxy if needed.
In order to use a socks proxy, you should either:
* start lava-slave with `--socks-proxy localhost:8081`
* edit `/etc/lava-dispatcher/lava-slave` and add
`SOCKS_PROXY="--socks-proxy localhost:8081"`
Custom CSS
==========
This adds the possibility to include a custom style sheet in a LAVA
instance for applying overall branding to the GUI.
Add the link to the CSS in the `settings.conf` and restart
`lava-server-gunicorn`:
```json
"BRANDING_CSS": "https://www.example.org/css/style.css",
```
Full list of changes since 2019.04
==================================
5e4a36956 jobs compress: create the output.yaml.size file
if needed.
412b73788 device-type: add SoCA9 from Schneider
aab54ce66 Revert "Auth refactoring part 1."
396c7342f logutils: test the module functions
15bb54214 logutils: only create the index when needed
ed60e521b Add a command line helper to compress logs
e19c9a2cf Read logs from either output.yaml or output.yaml.xz
41672feee job logs: replace TestJob.output_file by logutils
functions
c3bf98bd6 job: remove unused/uneeded job_file_present variable
c5c881347 Remove the archived_job_file function
1e32ed252 LAVA branding: add possibility to include custom CSS
417c8b14b release: add a script to remove old docker images
0e08fa08e Clean up obsolete conffiles causing piuparts failures
1e9778114 Present the codeclimate data in human readable format
2fa918740 schema: the job visibility is required.
2fbc72031 Order aliases by name when rendering
1a10f4903 scheduler.jobs.validate: fix return value
e7ce9d63c schema: allow admins to extend the white list of
context variable
a615696cb lava-schema: factorize and allow to check device config
0c68b065c schema: add a device schema into lava_common
1ac122efe Add fastboot support to the meson-g12a-sei510
56583c703 debian: remove unused or redondant files
9654c526a Remove permission object support from
GroupObjectPermission methods.
8d13f2def Change API to update single object permissions.
f365005b1 Small code review changes.
444404943 Add LAVAServerError exception tree.
7c533d5c4 Remove shortened codename support.
c991c3c5c Auth refactoring part 1.
c2acdbdd8 debian: remove unused py3dist overrides
dadce92a6 doc: fix sample job schema
c505f1322 schema: add missing uboot_needs_interrupt to the context
ccfbf2aed schema: fix boot.minimal by adding reset and auto_login
b17b24e94 schema: also validate the multinode context variable
0f13bc60a Change notification CharField types to TextField instead
4be183e2d unittests.sh was renamed to dispatcher-unittests.sh.
d6b642471 Partially revert 3d0ddb402 to fix some tables issues
ff613da21 device: improve the recent test jobs table
e21f2ddbb Use .exists() instead of len(.all())
bb3ca4525 XMLRPC API: fix crash in scheduler.aliases
85fc254b7 schema: fix remaining syntax error in the sample jobs
9cccb2e3a docs: Update docs for DeviceType alias
5b6424218 scheduler: add test for alias submissions
62c0e08ec scheduler/api: fix XML-RPC API for Aliases
ec74e9ed7 scheduler: avoid device type-alias name conflict
3d0ddb402 scheduler: modify alias to be 1-many relation
a6e9c6d2f lava-slave: add an option to use a socks proxy
7a18fc022 schema: allow uboot_extra_error_message in the context
711a20131 schema: fix definition of test definition option
"lava-signal"
b0088485c doc: fix emphasis
aefc87b26 Improve the README and use markdown for Gitlab rendering
a4b350516 fix pyocd boot action method to print out version of
binary used
c045401d1 Do not use minified version of JS and CSS files
55afae484 schema: add "new_connection" boot method
ed4eeda0a doc: add missing label
851b4bc33 version.py: fix git check when using worktrees
72b9dd18c Change the 'fastboot_sequence'
383525981 user commands: use run_cmd instead of run_command
b1c84a1d6 validate: allow to report invalid jobs to admins
82aa705b8 Fix multinode jobs validation
b10bc826f Use the static template function
d48ba02c6 Make USB/DFU/UDEV timeouts into Infra Errors
643180264 debian: list every minimun versions for buster
0dd5dc1d6 lava_common: remove unused dependency on pyzmq
f1c1372ca doc: fix syntax to match schema
759740e1c device-types: rpi: allow extra space for kernel image
3bd8f0d8a Fix file permissions
8139950b1 Remove USE_TEMPLATE_CACHE option
44c5e6ca5 scheduler: update rzn1d device-type to boot zImage
3fced42f5 Fix the dtb/ramdisk addresses
750d8f298 Document the "command" action block
0447e8327 doc: replace lava-tool by lavacli whenever possible
8c73a28ed Update packaging for dashboard removal
1c6be7f71 Remove leftover from v1 dashboard
cc7e86fa7 Remove dashboard_app
c4a4c3e3b device-type: add da850-lcdk
3a16ca954 Remove admin_notifications field
1a20620be CI: do not build anymore multiarch images
796c98eb3 CI: fix tag selection
cc2bc7509 add systemd support for lava-os-build
a7c483bdb Add overriding resets_after_flash to support newer
FRDM-K64F boards
537773c9e TestJob duration: return only seconds (and not
miliseconds)
5ae7f5a1a Add a helper script for Debian release
4b2c07da7 Fix broken seealso syntax in docs
447bc9344 doc/v2/timeouts: fix http-download action name typos
Steve, for the LAVA team
--
Steve McIntyre steve.mcintyre(a)linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
Hi folks,
The 2019.04 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are available at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and are
available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=======================
Device-types
============
New device-types
-----------------
New supported devices:
* A13-OLinuXino-MICRO (Allwinner)
* AM65x-GPEVM (aka am6) (TI)
* ar9331-dpt-module (DPTechnics)
* BeagleBone Black
* dove cubebox (Marvell)
* i.MX23 (Olimex)
* i.MX27-phytec-phycard-s-rdk (PHYTEC)
* i.MX28-duckbill (EnOcean)
* i.MX53-qsrb
* i.MX6ul-pico-hobbit
* i.MX6dl-riotboard (RIoTboard)
* meson-g12a-sei510 (Amlogic)
* meson-g12a-x96-max (Amlogic)
* SoCFPGA Cyclone V SoCrates (Altera)
cubietruck
----------
The bootloader prompt for `cubietruck` has been updated from `"sun7i"`
to `"sun7i# "`. In fact, the shorter form was also matching the dtb
filename, making some job unreliable.
qemu
----
If one or more test actions are included in a qemu job, LAVA will
create a disk image and will attach it to qemu.
Some emulated devices have no bus available for attaching this image
(like cubieboard). Some emulated devices have an available bus but
qemu is unable to attach to it due to the selected architecture (like
vexpress).
For these boards, you need to set `guestfs_interface` to `None` and
add a device with `drive=lavatest` in the job `context`:
context:
extra_options:
- -device virtio-bk-device,drive=lavatest
guestfs_interface: None
rzn1
----
`rz1` has been renamed to `rzn1d` to match OpenEmbedded board name.
Boot methods
============
Barebox
-------
A new boot method called `barebox` has been contributed. This method,
allows to boot boards using the `barebox` bootloader. Currently, the
resources are deployed using `tftp`.
A brebox job would look like:
actions:
- deploy:
to: tftp
kernel:
url: http://.../zImage
nfsrootfs:
url: http://.../jessie-armhf-nfs.tar.gz
compression: gz
dtb:
url: http://.../am335x-bone.dtb
…
- boot:
method: barebox
commands: nfs
prompts:
- 'root@jessie:'
…
The following device-types support barebox:
* ar9331-dpt-module (DPTechnics)
* BeagleBone Black
* dove cubebox (Marvell)
* i.MX23 (Olimex)
* i.MX27-phytec-phycard-s-rdk (PHYTEC)
* i.MX28-duckbill (EnOcean)
* i.MX53-qsbr
* i.MX6dl-riotboard (RIoTboard)
* SoCFPGA Cyclone V SoCrates (Altera)
dfu
---
Allow to use u-boot based dfu instead of the hardware one
On some boards, the hardware dfu serial number is set to 0000000. If
the board provides two boot media (NOR and NAND for example), we can
have a recovery u-boot on one media that will be used to flash the
second media using dfu.
actions:
- deploy:
to: tmpfs
images:
sf_uboot0:
image_arg: --alt sf_uboot0 --download {sf_uboot0}
url: http://.../u-boot-ddr.itb
- boot:
namespace: dfu
method: dfu
Grub
----
When booting with grub, users can now ask to serve the filesystem
from NBD instead of ramdisk or NFS.
- boot:
method: grub
commands: nbd
Minimal
-------
By default LAVA will reset the board power when executing the minimal
boot action. Users can skip this step by adding ``reset: false``. This
can be useful when testing bootloader in interactive tests and then
booting to the OS.
- boot:
method: minimal
reset: false
prompts:
- 'root@debian:~#'
- '/ #'
kexec
-----
The `kexec` boot method has been fixed. Thanks to this support, users
can test this kernel functionality:
- boot:
method: kexec
kernel: /boot/vmlinuz-3.16.0-4-amd64
initrd: /boot/initrd.img-3.16.0-4-amd64
command: /sbin/kexec
kernel-config: /boot/config-3.16.0-4-amd64
boot_message: Starting new kernel
options:
- "--reuse-cmdline"
auto_login:
login_prompt: "login:"
username: root
REST API
========
A new endpoint has been added to list `workers` at
`/api/v0.1/workers`.
Some fields are now using human readable names instead of opaque
constants:
* device:
* health
* state
* device-type:
* health_denominator
* TestJob:
* health
* state
* visibility
* submitter
Job definition schema
=====================
Web UI
------
When submitting or resubmitting a job from the web interface, the new
schema validator will be used. Any errors found by the new schema
validator will be reported as a warning.
Admin helper
------------
A new helper has been created for admins to validate old job
definitions on their local instance:
lava-server manage jobs validate --newer-than 1d
lava-server manage jobs validate --newer-than 1d --submitter lava-health
Job context
-----------
The schema validator is now checking the content of the `context`
dictionary. Only the following keys are now allowed:
* `arch`, `boot_console`, `boot_root`, `cpu`, `extra_options`, `guestfs_driveid`, `guestfs_interface`, `guestfs_size`, `machine`, `memory`, `model`, `monitor`, `netdevice`, `serial`, `vga`
* `bootloader_prompt`, `console_device`, `extra_kernel_args`, `extra_nfsroot_args`, `kernel_loglevel`, `kernel_start_message`, `lava_test_results_dir`, `menu_interrupt_prompt`, `mustang_menu_list`, `test_character_delay`, `tftp_mac_address`
Jobs using keys that are not listed in this list will be rejected.
If you think that some more keys should be whitelisted, please send a
patch.
Docker
======
Debian slim
-----------
The `lava-server` and `lava_dispatcher` docker images are now based on
`debian:stretch-slim`. This will reduce the size of the images
slightly.
lavaserver uid
--------------
In order to prevent any future issues, the `uid` and `gid` of the
`lavaserver` user has been fixed to `200`.
If you are upgrading from a previous lava-server docker image and
keeping the data (job logs, device dictionaries, ...) you will need to
change the owner of the files and directory:
chown -R lavaserver:lavaserver /etc/lava-server/dispatcher-config
chown -R lavaserver:lavaserver /var/lib/lava-server/
Full list of changes
====================
ec8e42c5f schema: vadidate context dictionnary
2e6964c17 device-types: fix board name imx53-qsbr -> imx53-qsrb
52e55a805 Tweak the reprepro-release.sh script again
73ad6ef10 Update the home page contents
d929cc8f9 Upgrade jQuery to v3.4.0 after the last security issue
8dd24244e Revert "device-types: base: detect TFTP ERROR: File not
found"
215a9ca04 Revert "device-types: use array instead of sting
concatenation"
976541d3a Update docs, adding more references to apt.lavasoftware.org
d0e9d7279 GitHelper: when failing, print git command stdout/stderr
e5671dbf1 u-boot-dfu tests: skip if dfu-util is not available
0b660a124 lava shell helpers: fix shellcheck warnings
7f02689c2 Allow to use u-boot based dfu instead of the hardware one
f81e83f31 qemu: Add an ID to the qemu storage
e063102e2 Convert offset and limit query parameters to ints
6363d1020 results: use yaml CDumper instead of the python dumper
d259def87 Fix documentation about template caching
2dbb0e059 device-types: use array instead of sting concatenation
0b3f671bf Fix crash when validating an invalid multinode job
0b6550d6f sample jobs: improve syntax
7c38216ea Schema: remove "live" argument
7e0a8e415 Fix file permissions
562be7cc6 Remove unused allow_modify from grub-nbd sample job
6f25a8086 device-types: add socfpga-cyclone5-socrates device
660b5b126 device-types: add ar9331-dpt-module device
81fa52c07 device-types: add dove-cubox device
65dacd712 device-types: add imx6dl-riotboard device
4566a4639 device-types: add imx53-qsbr device
7b1d2d0b9 device-types: add imx27-phytec-phycard-s-rdk device
2735228e1 device-types: add imx28-duckbill device
91abf4c2c device-types: add imx23-olinuxino device
efec57256 sample_jobs: add barebox job definitions
7b8e78b3c device-types: add parameters section to barebox type
b8102f433 device-types: add helper for barebox test and use on
imx6ul-pico-hobbit
ca800a7b6 xmlrpc: do not raise if jobs.list is empty
705d0aeeb schema: improve inline test definition schema
f5e06bf7d lava-schema: print yaml syntax errors
000420042 schema: allow for auto_login in fastboot and kexec
29708cbb0 schema: allow for no compression
883e24be4 schema: enforce that every actions of a multinode use roles
e9fe783bf Show validation warnings when (re)submitting jobs in the web
ui
c3f8993fe Add NBD support to grub
07a693eaf device-types: minnowboard-common: permit to override
grub_autoboot_prompt
52c9f1ce8 device-types:: allow to change bootloader prompt at device
level
b3864a998 schema: accept zero MultiNode role count
78de67ce0 schema: accept test action parameters
5143b42f3 device-types: add sun5i-a13-olinuxino-micro
7403ddfd4 Remove all links to lavapdu and improve PDUDaemon docs
613038ccd doc: fix typo on --passwd argument
c56ab79e5 schema: add deploy.overlay module
5043f5e83 doc: make the message toward 3rd party packaging more
friendly
0140f42f3 Remove unsed argument
86a923cde Fix version/arch computation in docker debian-slim variants
79d239060 CI: use new sast template provided by gitlab
fce87ba18 cubietruck: use a longer bootloader prompt
83de037f0 Improve sql efficiency when generating lab report
4bab2f601 Improve "lava-server manage check"
714cdb587 Move back adduser call from postinst.py to
lava-server.install
668d132c6 device-types: base: detect TFTP ERROR: File not found
5bf642e3b scheduler: added AM6 device type
eb784fa69 Remove the top-level links from the docs
22f816bbb Add extra doc template code to add rel="canonical" links
95b43d160 boot minimal: allow to skip reseting the device
54f00eea7 Fix kexec boot failure
4819cb9db lava-test-raise: fix shell syntax
d718a4a9d device-type: add meson-g12a-x96-max
0f29a3e9e device-type: add meson-g12a-sei510
386c29ea0 device-type: add new meson-g12-common
affccf663 auto login: fix the prompts list after login
bcb243c65 debian: remove dependencies to unused jquery libs
eece2782b CI: skip debian specific tests if dpkg-query is not
installed
f2fdfa0f9 Improve error message
c974a8859 scheduler: rename rzn1 to rzn1d
b2cebf9a1 Only deploy the LAVA docs on a tag
a7a3f1f2f schema: allow namespace in user commands
f5b6edf73 Use enumerate whenever possible
f44347ce9 Remove "-master" (branch name) in docker tags
e09d68f19 Fix argument typo.
7cd503803 lava_scheduler_app: add barebox test
9ba1c02eb device-types: add imx6ul-pico-hobbit device
9e7a7ecd2 device-types: add base-barebox for typical barebox setup
6fda3991d dispatcher: add barebox boot action
1a6bca180 REST API: translate dictionary values to human readable form
e3b6fde17 Remove /etc/lava-dispatcher/lava-dispatcher.conf
2b0c7569e Upgrade bootstrap to v3.4.1
4cf162244 Use bootstrap minified css file
10396e6a0 release: add the script to push docker containers to docker
hub
2a7e17ab6 check: fix the check for systemd existence
bf4dc683e check: fix a crash when the uid is unknown
443a8b87b Add validate helper to manage jobs support
58eb62334 Remove unused imports
f51d9d000 rest-api: also list workers
2e7cae991 Add --host argument to pg_isready command call
b5090e4d9 debian/lava-dispatcher.dirs: remove usr/sbin
936541ff6 Fix typo in lava-modules.conf install path
Steve, for the LAVA team
--
Steve McIntyre steve.mcintyre(a)linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
