Hi folks,
The 2019.07 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=======================
Device-types
============
New device-types
----------------
New supported devices:
* meson-gxl-s805x-libretech-ac (Amlogic)
* meson-gxl-s905d-p230 (Amlogic)
* meson-gxl-s905x-p212 (Amlogic)
* meson8b-ec100 (Amlogic)
* sun50i-h6-pine-h64
ox820-cloudengines-pogoplug-series-3
------------------------------------
Some ARM defconfig changes made the kernel bigger and the ox820 device
cannot boot any more. Update the ramdisk address to **0x61000000**
(was **0x60e00000**).
extra_nfsroot_args
------------------
Some device-types (`jetson-tk1`, `juno`, `qcom-qdf2400`, `rzn1d`,
`synquacer` and `tc2`) were not allowing the job submitter to set
`extra_nfsroot_args` in the job context.
Job submitters can now pass extra nfs rootfs arguments in the job
context for these devices.
context:
extra_nfsroot_args: ",wsize=65536 nfsrootdebug"
Database
========
Migrations
----------
This new version creates several database migrations:
* add `ExtendedUser.table_length` field
* add new tables: `GroupDeviceTypePermission` and `GroupDevicePermission`
* add custom permissions for Device: `view_device`, `submit_to_device`,
`admin_device`
* add custom permissions for DeviceType: `view_devicetype`,
`submit_to_devicetype`, `admin_devicetype`
* remove fields from DeviceType: `owners_only`
* remove fields from Device: `user`, `group`, `is_public`
* remove fields from TestJob: `user`, `group`, `visibility`
* delete table: `DefaultDeviceOwner`
These may incur a small amount of downtime during the upgrade process
as database migrations happen. This may take a long time if there is a
large number of records in the ``TestJob`` table.
A goal for the LAVA development team is to reduce the frequency of
database migrations to reduce this cost as much as possible. This
release also includes a data migration and adding new records in the
newly created tables so that authorization refactoring does not change
any existing user-visible behaviour.
Authorization refactoring
=========================
Due to scaling problems with the previous authentication model, the
way LAVA controls access to its most important entities (DeviceType,
Device and TestJob) has been completely revamped.
Previously, LAVA used flags like `owners_only`, `user`, `group`,
`visibility`. On large LAVA installations these were causing
significant database overhead. They have now been removed and replaced
by a new design. Two new tables have been added:
`GroupDeviceTypePermission` and `GroupDevicePermission`. The
permission model has been moved to Group based control and
administrators can control those through the Django admin interfaces
for ``DeviceType`` and ``Device`` records, or via the respective
XMLRPC calls. The data migration will convert records across from the
old auth model to the new model, keeping equivalent permissions.
The new permission model is inheritable by design so any permissions
applied to device types will also be applied to devices belonging to
the particular device type, and all permissions applied to a device
will be relevant for permissions on test jobs running on that
particular device. Further information can be found in the LAVA
documentation.
Table size
==========
The LAVA web interface displays a lot of tables. The size of the table
is configurable at instance level by the admin since LAVA
[2019.06](2019.06). There is a default table length defined in the
system.
In this release, users can now also configure the default table length
to use when logged in. This can be set by the user in their profile
page at `https://<host>/me/`.
Steve, for the LAVA team.
--
Steve McIntyre steve.mcintyre(a)linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
Hi folks,
The 2019.07 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and will be available at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=======================
Device-types
============
New device-types
----------------
New supported devices:
* sun50i-h6-orangepi-one-plus (Allwinner)
* sun8i-a33-olinuxino (Olimex)
* TI CC13X2 LaunchPad (TI)
cmsis-dap
---------
In order to fix some infrequent issues on specific hardware and boards,
LAVA will now explicitly perform a sync on the directory before umounting
it.
depthcharge
-----------
### extra kernel arguments
Admins and users can now both define some extra kernel arguments in
device-type templates and in the job context.
Use `base_kernel_args` in device-type templates and `extra_kernel_args` in
job definitions.
### LZMA compression
Enable LZMA compression for arm64 kernel Image with Depthcharge.
docker
------
Secondary ssh connection is now supported for the docker device-type. Look
at the [documentation](
https://docs.lavasoftware.org/lava/pipeline-writer-secondary.html#secure-sh…)
for more information.
lxc
---
Under some circumstances, LAVA could leak the lxc rootfs in
`/var/lib/lxc/`. LAVA will no longer leak such rootfs.
We advice admins to have a look at `/var/lib/lxc` and to remove unused
rootfs as they can use a significant amount of disk space.
OpenOCD
-------
Admins can set `board_selection_cmd` in the device-type template to specify
a command to pass the board id/serial number to OpenOCD.
See OpenOCD documentation for details on the command to set the serial
number for the interface your device type is using.
Docker
======
Dockerfiles
-----------
The repository [lava/pkg/docker](
https://git.lavasoftware.org/lava/pkg/docker), hosting the Dockerfiles has
been merged into the [lava/lava](https://git.lavasoftware.org/lava/lava)
repository.
This merge will allow (in a near future) to build the docker images from
sources.
entrypoint.d
------------
When starting `lava-server` or `lava-dispatcher` docker images, admin can
execute custom scripts by adding them into `/root/entrypoint.d/` The
scripts will be executed right before the start of the services.
In this release the support for custom script has been fixed.
performances
------------
When running gunicorn in docker container, the service will now use
`/dev/shm` for storing heartbeat artifacts.
By default, gunicorn will use /tmp which is not mounted in docker
containers. `/dev/shm` is guaranteed to be available and an in-memory
filesystem.
Using a non-memory file system can lead to intermittent hang.
See [gunicorn in docker](
https://pythonspeed.com/articles/gunicorn-in-docker/) for more information.
permissions
-----------
When starting `lava-logs` or `lava-server-gunicorn` the entrypoint will now
check that the files under `/etc/lava-server/dispatcher-config/` and
`/var/lib/lava-server/default/media/` belong to the `lavaserver` user and
group.
The docker container will refuse to start if that's not the case as this
can lead to crashes or strange behaviors.
lava-publisher
==============
The lava-publisher sockets will now automatically send ping messages thanks
to the [ZMTP protocol](https://rfc.zeromq.org/spec:37/ZMTP/).
If the connection is lost, the [heartbeating protocol](
https://rfc.zeromq.org/spec:37/ZMTP/#connection-heartbeating) will detect
the issue and try to reconnect.
Error reporting
===============
In order to track crashes of the LAVA services, admins can setup [sentry](
https://sentry.io) in `/etc/lava-server/settings.conf`:
```json
"SENTRY_DSN": "http://<public_key>@<sentry>/<project-id>"
```
You should also install `sentry-sdk` from [pypi](
https://pypi.org/project/sentry-sdk/) with:
```shell
apt-get install python3-pip
python3 -m pip install sentry-sdk==0.10.2
```
The lava-server docker image comes with the sentry sdk already installed.
Rémi, for the LAVA team.
--
Rémi Duraffort
LAVA Team, Linaro
Hello,
following the release of LAVA 2019.06, we released lavacli v0.9.7
This release brings only one new feature: get/set worker environment
In order to use this feature, you need LAVA >= 2019.06
Then you can:
lavacli workers env get <hostname>
lavacli workers env set <hostname> <env.yaml>
As usual, lavacli could be installed from:
* pypi (https://pypi.org/project/lavacli/)
* debian (https://tracker.debian.org/pkg/lavacli) (coming soon).
Cheers
--
Rémi Duraffort
LAVA Team, Linaro
Hi folks,
The 2019.06 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are available at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=======================
Device-types
============
New device-types
----------------
New supported devices:
* dragonboard-845c
* meson-g12b-odroid-n2 (Amlogic)
* ox820-cloudengines-pogoplug-series-3 (CloudEngines)
* stm32mp157c-dk2 (STMicroelectronics)
* sun50i-a64-bananapi-m64 (Allwinner)
* sun7i-a20-cubieboard2 (Allwinner)
* sun8i-a83t-bananapi-m3 (Allwinner)
* sun8i-r40-bananapi-m2-ultra (Allwinner)
docker
------
Add an option to start docker device as privileged. Add `{% set
docker_privileged = True %}` to the device dictionary.
qemu
----
Add support for `nios2` and `xtensa` architectures.
ssh
---
The ssh username can now be set in the device dictionary.
Boot methods
============
OpenOCD
-------
A new boot method called `openocd` has been contributed. This method,
allows to flash and boot boards using `openocd`.
An openocd job would look like:
- deploy:
to: tmpfs
images:
binary:
url: http://.../zephyr.elf
openocd_script:
url: http://.../cc3220_xds110.cfg
- boot:
method: openocd
- test:
monitors:
- name: tests
start: (tc_start\(\)|starting .*test|BOOTING ZEPHYR OS)
end: PROJECT EXECUTION SUCCESSFUL
pattern: (?P<result>(PASS|FAIL))\s-\s(?P<test_case_id>\w+)\r\n
fixupdict:
PASS: pass
FAIL: fail
For the moment, only the `cc3220SF` board supports this method.
Test shell
==========
result strings
---------------
In LAVA versions prior to 2019.01 both lower and upper case results
were accepted. This behavior was changed in 2019.01 to only accept
lower case.
In order to keep compatibility among test jobs, the upper case result
strings (`PASS`, `FAIL`, `SKIP` and `UNKNOWN`) are now valid again.
environment
-----------
A new variable, called `environment` has been added to the device
dictionaries. These can be used, for example, to describe physical
hardware connections between the DUT and interfaces on the worker or
other addressable hardware.
For ease of use, LAVA will directly export the content of the
`device_info`, `environment`, `static_info` and `storage_info`
dictionaries into the test shell environment. The dictionaries and
lists will be unrolled, for example:
{% set environment = {"RELAY_ADDRESS": "10.66.16.103", "REMOTE_SERIAL_PORT": "/dev/ttyUSB2"} %}
{% set static_info = [{"board_id": "S_NO81730000"}, {"board_id": "S_NO81730001"}] %}
{% set storage_info = [{'SATA': '/dev/disk/by-id/ata-ST500DM002-1BD142_W3T79GCW'}] %}
will become:
export RELAY_ADDRESS='10.66.16.103'
export REMOTE_SERIAL_PORT='/dev/ttyUSB2'
export LAVA_STATIC_INFO_0_board_id='S_NO81730000'
export LAVA_STATIC_INFO_1_board_id='S_NO81730001'
export LAVA_STORAGE_INFO_0_SATA='/dev/disk/by-id/ata-ST500DM002-1BD142_W3T79GCW'
REST API
========
Filtering has been activated in the REST API for 'devices',
'device-types' and 'jobs'. See the
[documentation](https://docs.lavasoftware.org/lava/data-export.html#filterin…
for the details.
Table size
==========
The LAVA web interface displays a lot of tables. The size of the table
is now configurable at instance level by the admin.
Update `settings.conf` and restart `lava-server-gunicorn`:
"DEFAULT_TABLE_LENGTH": 50,
By default, every table will display 25 elements.
lava-slave
==========
Socks proxy
-----------
When connecting with `lava-master` and `lava-logs`, `lava-slave` can
now use a socks proxy if needed.
In order to use a socks proxy, you should either:
* start lava-slave with `--socks-proxy localhost:8081`
* edit `/etc/lava-dispatcher/lava-slave` and add
`SOCKS_PROXY="--socks-proxy localhost:8081"`
The support is now complete.
Job definition schema
=====================
Job context
-----------
The schema validator is now checking the content of the `context`
dictionary for both single and multinode jobs.
A new set of keys has been added: `boot_character_delay`,
`booti_dtb_addr`, `booti_kernel_addr`, `booti_ramdisk_addr`,
`bootm_dtb_addr`, `bootm_kernel_addr`, `bootm_ramdisk_addr`,
`bootz_dtb_addr`, `bootz_kernel_addr`, `bootz_ramdisk_addr`.
Only the following keys are now allowed: * `arch`, `boot_console`,
`boot_root`, `cpu`, `extra_options`, `guestfs_driveid`,
`guestfs_interface`, `guestfs_size`, `machine`, `memory`, `model`,
`monitor`, `netdevice`, `serial`, `vga` * `booti_dtb_addr`,
`booti_kernel_addr`, `booti_ramdisk_addr`, `bootm_dtb_addr`,
`bootm_kernel_addr`, `bootm_ramdisk_addr`, `bootz_dtb_addr`,
`bootz_kernel_addr`, `bootz_ramdisk_addr` * `boot_character_delay`,
`bootloader_prompt`, `console_device`, `extra_kernel_args`,
`extra_nfsroot_args`, `kernel_loglevel`, `kernel_start_message`,
`lava_test_results_dir`, `menu_interrupt_prompt`, `mustang_menu_list`,
`test_character_delay`, `tftp_mac_address`,
`uboot_extra_error_message`, `uboot_needs_interrupt`
Jobs using keys that are not listed in this list will be rejected.
If you think that some more keys should be whitelisted, talk to us.
XML-RPC
=======
Two functions called `scheduler.workers.get_env` and
`scheduler.workers.set_env` have been introduced. These functions
allow users to get and set the environment for a given dispatcher.
The corresponding command line has been added to `lavacli` v0.9.7:
$ lavacli workers env get dispatcher01
$ lavacli workers env set dispatcher01 env.yaml
Steve, for the LAVA team
--
Steve McIntyre steve.mcintyre(a)linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
Hi folks,
We found a significant bug in our latest 2019.05 release, affecting
the download of large compressed log files. We have released a hotfix
update for this issue, tagged as 2019.05post1.
.deb packages have been built in GitLab CI and are available at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
will be available shortly from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=======================
Full list of changes since 2019.04
==================================
8b309eadb Pass iterator to StreamingHttpReponse
Steve, for the LAVA team
--
Steve McIntyre steve.mcintyre(a)linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
Hello,
following the release of LAVA 2019.05, we released lavacli v0.9.6
This release brings some improvements:
* "jobs cancel" can now cancel multiple jobs
* "utils templates render" accepts a context and can trim the output
* add a timeout to "jobs wait" command
In order to use the new aliases introduced/fixed in LAVA 2019.05, lavacli
v0.9.6 is required.
As usual, lavacli could be installed from:
* pypi (https://pypi.org/project/lavacli/)
* debian (https://tracker.debian.org/pkg/lavacli) (coming soon).
Cheers
--
Rémi Duraffort
LAVA Team, Linaro
Hi folks,
The 2019.05 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are available at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
will be available shortly from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=======================
Device-types
============
Aliases
-------
The device-type aliases feature has been totally redesigned. You can
now define multiple aliases for a given device-type. In a job
definition, users can use either the device-type or any of its
corresponding aliases.
New device-types
----------------
New supported devices:
* da850-lcdk (davinci)
* SoCA9 (Schneider)
frdm-k64f
---------
It is now possible to override the setting `resets_after_flash` in the
device dictionary, to support newer firmwares.
meson-gl12
----------
Fix the dtb and ramdisk addresses:
* dtb: **0x01070000** (was **0x0107f000**)
* ramdisk: **0x08000000** (was **0x04000000**)
RPi
---
Move the ramdisk load address to **0x02200000** (was **0x02100000**)
to allow more space for the kernel image and avoid them overlapping.
rzn1d
-----
Allow to use zImage.
Database
========
Migrations
----------
This new version creates several database migrations to:
* remove `TestJob.admin_notifications` field
* improve `DeviceType` aliases
* replace some CharField types with TextField to remove unnecessary
size limitations
These may incur a small amount of downtime during the upgrade process
as database migrations happen. A goal for the LAVA development team is
to reduce the frequency of database migrations to reduce this cost as
much as possible. The next **planned** set of database migrations is
in the 2019.08 release, in August.
dashboard_app
-------------
The dashboard_app (holding v1 results) has been fully removed,
including:
* migrations
* database model
* xmlrpc calls
Docker
======
multiarch
---------
Due to some limitations in the docker registry embedded in GitLab, we
are no longer building and providing multiarch docker images for every
commit. Multiarch images will still be available on [docker
hub](https://hub.docker.com/u/lavasoftware/) for every release.
lava-dispatcher
---------------
The dispatcher docker image now includes `docker-ce-cli`. This will
allow lava to control docker containers from within a lava-dispatcher
container, helping admins to participate in lavafed.
Server performances
===================
Template caching
----------------
The `USE_TEMPLATE_CACHE` setting variable has been removed. On newer
Django versions, this option is no longer useful and might even cause
problems.
Compressing job logs
--------------------
Support has been added for compressing log files on the server, using
xz (lzma) compression. Logs will be stored uncompressed as jobs are
running, but they may later be compressed by admins to reduce the
amount of space needed.
The `lava-server manage jobs compress` command
includes options to allow for management of this. Admins can use this
in cronjobs to automatically compress job logs older than 30 days, or
similar (`lava-server manage jobs compress --older-than 30d --slow`).
LAVA will automatically decompress old compressed logs on the fly if
they are ever needed again in future.
Job definition schema
=====================
Admin helper
------------
The `lava-server manage jobs validate` command can now send a mail to
the admin when an invalid job is found.
To make use of this, admins can setup a regular cronjob that will call:
```shell
lava-server manage jobs validate --mail-admins
```
Job context
-----------
The schema validator is now checking the content of the `context`
dictionary for both single and multinode jobs.
Two new allowed keys have been added: `uboot_extra_error_message` and
`uboot_needs_interrupt`
Only the following keys are now allowed:
* `arch`, `boot_console`, `boot_root`, `cpu`, `extra_options`, `guestfs_driveid`, `guestfs_interface`, `guestfs_size`, `machine`, `memory`, `model`, `monitor`, `netdevice`, `serial`, `vga`
* `bootloader_prompt`, `console_device`, `extra_kernel_args`, `extra_nfsroot_args`, `kernel_loglevel`, `kernel_start_message`, `lava_test_results_dir`, `menu_interrupt_prompt`, `mustang_menu_list`, `test_character_delay`, `tftp_mac_address`, `uboot_extra_error_message`, `uboot_needs_interrupt`
Jobs using keys that are not listed in this list will be rejected.
If you think that some more keys should be whitelisted, talk to us.
Extending job context
---------------------
Admins can extend the white list by updating `EXTRA_CONTEXT_VARIABLES`
in the settings:
```json
"EXTRA_CONTEXT_VARIABLES": ["custom_variable1", "variable_2"]
```
Don't forget to restart `lava-server-gunicorn` if you make changes
here.
lava-slave
==========
Socks proxy
-----------
When connecting with `lava-master` and `lava-logs`, `lava-slave` can
now use a socks proxy if needed.
In order to use a socks proxy, you should either:
* start lava-slave with `--socks-proxy localhost:8081`
* edit `/etc/lava-dispatcher/lava-slave` and add
`SOCKS_PROXY="--socks-proxy localhost:8081"`
Custom CSS
==========
This adds the possibility to include a custom style sheet in a LAVA
instance for applying overall branding to the GUI.
Add the link to the CSS in the `settings.conf` and restart
`lava-server-gunicorn`:
```json
"BRANDING_CSS": "https://www.example.org/css/style.css",
```
Full list of changes since 2019.04
==================================
5e4a36956 jobs compress: create the output.yaml.size file
if needed.
412b73788 device-type: add SoCA9 from Schneider
aab54ce66 Revert "Auth refactoring part 1."
396c7342f logutils: test the module functions
15bb54214 logutils: only create the index when needed
ed60e521b Add a command line helper to compress logs
e19c9a2cf Read logs from either output.yaml or output.yaml.xz
41672feee job logs: replace TestJob.output_file by logutils
functions
c3bf98bd6 job: remove unused/uneeded job_file_present variable
c5c881347 Remove the archived_job_file function
1e32ed252 LAVA branding: add possibility to include custom CSS
417c8b14b release: add a script to remove old docker images
0e08fa08e Clean up obsolete conffiles causing piuparts failures
1e9778114 Present the codeclimate data in human readable format
2fa918740 schema: the job visibility is required.
2fbc72031 Order aliases by name when rendering
1a10f4903 scheduler.jobs.validate: fix return value
e7ce9d63c schema: allow admins to extend the white list of
context variable
a615696cb lava-schema: factorize and allow to check device config
0c68b065c schema: add a device schema into lava_common
1ac122efe Add fastboot support to the meson-g12a-sei510
56583c703 debian: remove unused or redondant files
9654c526a Remove permission object support from
GroupObjectPermission methods.
8d13f2def Change API to update single object permissions.
f365005b1 Small code review changes.
444404943 Add LAVAServerError exception tree.
7c533d5c4 Remove shortened codename support.
c991c3c5c Auth refactoring part 1.
c2acdbdd8 debian: remove unused py3dist overrides
dadce92a6 doc: fix sample job schema
c505f1322 schema: add missing uboot_needs_interrupt to the context
ccfbf2aed schema: fix boot.minimal by adding reset and auto_login
b17b24e94 schema: also validate the multinode context variable
0f13bc60a Change notification CharField types to TextField instead
4be183e2d unittests.sh was renamed to dispatcher-unittests.sh.
d6b642471 Partially revert 3d0ddb402 to fix some tables issues
ff613da21 device: improve the recent test jobs table
e21f2ddbb Use .exists() instead of len(.all())
bb3ca4525 XMLRPC API: fix crash in scheduler.aliases
85fc254b7 schema: fix remaining syntax error in the sample jobs
9cccb2e3a docs: Update docs for DeviceType alias
5b6424218 scheduler: add test for alias submissions
62c0e08ec scheduler/api: fix XML-RPC API for Aliases
ec74e9ed7 scheduler: avoid device type-alias name conflict
3d0ddb402 scheduler: modify alias to be 1-many relation
a6e9c6d2f lava-slave: add an option to use a socks proxy
7a18fc022 schema: allow uboot_extra_error_message in the context
711a20131 schema: fix definition of test definition option
"lava-signal"
b0088485c doc: fix emphasis
aefc87b26 Improve the README and use markdown for Gitlab rendering
a4b350516 fix pyocd boot action method to print out version of
binary used
c045401d1 Do not use minified version of JS and CSS files
55afae484 schema: add "new_connection" boot method
ed4eeda0a doc: add missing label
851b4bc33 version.py: fix git check when using worktrees
72b9dd18c Change the 'fastboot_sequence'
383525981 user commands: use run_cmd instead of run_command
b1c84a1d6 validate: allow to report invalid jobs to admins
82aa705b8 Fix multinode jobs validation
b10bc826f Use the static template function
d48ba02c6 Make USB/DFU/UDEV timeouts into Infra Errors
643180264 debian: list every minimun versions for buster
0dd5dc1d6 lava_common: remove unused dependency on pyzmq
f1c1372ca doc: fix syntax to match schema
759740e1c device-types: rpi: allow extra space for kernel image
3bd8f0d8a Fix file permissions
8139950b1 Remove USE_TEMPLATE_CACHE option
44c5e6ca5 scheduler: update rzn1d device-type to boot zImage
3fced42f5 Fix the dtb/ramdisk addresses
750d8f298 Document the "command" action block
0447e8327 doc: replace lava-tool by lavacli whenever possible
8c73a28ed Update packaging for dashboard removal
1c6be7f71 Remove leftover from v1 dashboard
cc7e86fa7 Remove dashboard_app
c4a4c3e3b device-type: add da850-lcdk
3a16ca954 Remove admin_notifications field
1a20620be CI: do not build anymore multiarch images
796c98eb3 CI: fix tag selection
cc2bc7509 add systemd support for lava-os-build
a7c483bdb Add overriding resets_after_flash to support newer
FRDM-K64F boards
537773c9e TestJob duration: return only seconds (and not
miliseconds)
5ae7f5a1a Add a helper script for Debian release
4b2c07da7 Fix broken seealso syntax in docs
447bc9344 doc/v2/timeouts: fix http-download action name typos
Steve, for the LAVA team
--
Steve McIntyre steve.mcintyre(a)linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
Hi folks,
The 2019.04 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are available at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and are
available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=======================
Device-types
============
New device-types
-----------------
New supported devices:
* A13-OLinuXino-MICRO (Allwinner)
* AM65x-GPEVM (aka am6) (TI)
* ar9331-dpt-module (DPTechnics)
* BeagleBone Black
* dove cubebox (Marvell)
* i.MX23 (Olimex)
* i.MX27-phytec-phycard-s-rdk (PHYTEC)
* i.MX28-duckbill (EnOcean)
* i.MX53-qsrb
* i.MX6ul-pico-hobbit
* i.MX6dl-riotboard (RIoTboard)
* meson-g12a-sei510 (Amlogic)
* meson-g12a-x96-max (Amlogic)
* SoCFPGA Cyclone V SoCrates (Altera)
cubietruck
----------
The bootloader prompt for `cubietruck` has been updated from `"sun7i"`
to `"sun7i# "`. In fact, the shorter form was also matching the dtb
filename, making some job unreliable.
qemu
----
If one or more test actions are included in a qemu job, LAVA will
create a disk image and will attach it to qemu.
Some emulated devices have no bus available for attaching this image
(like cubieboard). Some emulated devices have an available bus but
qemu is unable to attach to it due to the selected architecture (like
vexpress).
For these boards, you need to set `guestfs_interface` to `None` and
add a device with `drive=lavatest` in the job `context`:
context:
extra_options:
- -device virtio-bk-device,drive=lavatest
guestfs_interface: None
rzn1
----
`rz1` has been renamed to `rzn1d` to match OpenEmbedded board name.
Boot methods
============
Barebox
-------
A new boot method called `barebox` has been contributed. This method,
allows to boot boards using the `barebox` bootloader. Currently, the
resources are deployed using `tftp`.
A brebox job would look like:
actions:
- deploy:
to: tftp
kernel:
url: http://.../zImage
nfsrootfs:
url: http://.../jessie-armhf-nfs.tar.gz
compression: gz
dtb:
url: http://.../am335x-bone.dtb
…
- boot:
method: barebox
commands: nfs
prompts:
- 'root@jessie:'
…
The following device-types support barebox:
* ar9331-dpt-module (DPTechnics)
* BeagleBone Black
* dove cubebox (Marvell)
* i.MX23 (Olimex)
* i.MX27-phytec-phycard-s-rdk (PHYTEC)
* i.MX28-duckbill (EnOcean)
* i.MX53-qsbr
* i.MX6dl-riotboard (RIoTboard)
* SoCFPGA Cyclone V SoCrates (Altera)
dfu
---
Allow to use u-boot based dfu instead of the hardware one
On some boards, the hardware dfu serial number is set to 0000000. If
the board provides two boot media (NOR and NAND for example), we can
have a recovery u-boot on one media that will be used to flash the
second media using dfu.
actions:
- deploy:
to: tmpfs
images:
sf_uboot0:
image_arg: --alt sf_uboot0 --download {sf_uboot0}
url: http://.../u-boot-ddr.itb
- boot:
namespace: dfu
method: dfu
Grub
----
When booting with grub, users can now ask to serve the filesystem
from NBD instead of ramdisk or NFS.
- boot:
method: grub
commands: nbd
Minimal
-------
By default LAVA will reset the board power when executing the minimal
boot action. Users can skip this step by adding ``reset: false``. This
can be useful when testing bootloader in interactive tests and then
booting to the OS.
- boot:
method: minimal
reset: false
prompts:
- 'root@debian:~#'
- '/ #'
kexec
-----
The `kexec` boot method has been fixed. Thanks to this support, users
can test this kernel functionality:
- boot:
method: kexec
kernel: /boot/vmlinuz-3.16.0-4-amd64
initrd: /boot/initrd.img-3.16.0-4-amd64
command: /sbin/kexec
kernel-config: /boot/config-3.16.0-4-amd64
boot_message: Starting new kernel
options:
- "--reuse-cmdline"
auto_login:
login_prompt: "login:"
username: root
REST API
========
A new endpoint has been added to list `workers` at
`/api/v0.1/workers`.
Some fields are now using human readable names instead of opaque
constants:
* device:
* health
* state
* device-type:
* health_denominator
* TestJob:
* health
* state
* visibility
* submitter
Job definition schema
=====================
Web UI
------
When submitting or resubmitting a job from the web interface, the new
schema validator will be used. Any errors found by the new schema
validator will be reported as a warning.
Admin helper
------------
A new helper has been created for admins to validate old job
definitions on their local instance:
lava-server manage jobs validate --newer-than 1d
lava-server manage jobs validate --newer-than 1d --submitter lava-health
Job context
-----------
The schema validator is now checking the content of the `context`
dictionary. Only the following keys are now allowed:
* `arch`, `boot_console`, `boot_root`, `cpu`, `extra_options`, `guestfs_driveid`, `guestfs_interface`, `guestfs_size`, `machine`, `memory`, `model`, `monitor`, `netdevice`, `serial`, `vga`
* `bootloader_prompt`, `console_device`, `extra_kernel_args`, `extra_nfsroot_args`, `kernel_loglevel`, `kernel_start_message`, `lava_test_results_dir`, `menu_interrupt_prompt`, `mustang_menu_list`, `test_character_delay`, `tftp_mac_address`
Jobs using keys that are not listed in this list will be rejected.
If you think that some more keys should be whitelisted, please send a
patch.
Docker
======
Debian slim
-----------
The `lava-server` and `lava_dispatcher` docker images are now based on
`debian:stretch-slim`. This will reduce the size of the images
slightly.
lavaserver uid
--------------
In order to prevent any future issues, the `uid` and `gid` of the
`lavaserver` user has been fixed to `200`.
If you are upgrading from a previous lava-server docker image and
keeping the data (job logs, device dictionaries, ...) you will need to
change the owner of the files and directory:
chown -R lavaserver:lavaserver /etc/lava-server/dispatcher-config
chown -R lavaserver:lavaserver /var/lib/lava-server/
Full list of changes
====================
ec8e42c5f schema: vadidate context dictionnary
2e6964c17 device-types: fix board name imx53-qsbr -> imx53-qsrb
52e55a805 Tweak the reprepro-release.sh script again
73ad6ef10 Update the home page contents
d929cc8f9 Upgrade jQuery to v3.4.0 after the last security issue
8dd24244e Revert "device-types: base: detect TFTP ERROR: File not
found"
215a9ca04 Revert "device-types: use array instead of sting
concatenation"
976541d3a Update docs, adding more references to apt.lavasoftware.org
d0e9d7279 GitHelper: when failing, print git command stdout/stderr
e5671dbf1 u-boot-dfu tests: skip if dfu-util is not available
0b660a124 lava shell helpers: fix shellcheck warnings
7f02689c2 Allow to use u-boot based dfu instead of the hardware one
f81e83f31 qemu: Add an ID to the qemu storage
e063102e2 Convert offset and limit query parameters to ints
6363d1020 results: use yaml CDumper instead of the python dumper
d259def87 Fix documentation about template caching
2dbb0e059 device-types: use array instead of sting concatenation
0b3f671bf Fix crash when validating an invalid multinode job
0b6550d6f sample jobs: improve syntax
7c38216ea Schema: remove "live" argument
7e0a8e415 Fix file permissions
562be7cc6 Remove unused allow_modify from grub-nbd sample job
6f25a8086 device-types: add socfpga-cyclone5-socrates device
660b5b126 device-types: add ar9331-dpt-module device
81fa52c07 device-types: add dove-cubox device
65dacd712 device-types: add imx6dl-riotboard device
4566a4639 device-types: add imx53-qsbr device
7b1d2d0b9 device-types: add imx27-phytec-phycard-s-rdk device
2735228e1 device-types: add imx28-duckbill device
91abf4c2c device-types: add imx23-olinuxino device
efec57256 sample_jobs: add barebox job definitions
7b8e78b3c device-types: add parameters section to barebox type
b8102f433 device-types: add helper for barebox test and use on
imx6ul-pico-hobbit
ca800a7b6 xmlrpc: do not raise if jobs.list is empty
705d0aeeb schema: improve inline test definition schema
f5e06bf7d lava-schema: print yaml syntax errors
000420042 schema: allow for auto_login in fastboot and kexec
29708cbb0 schema: allow for no compression
883e24be4 schema: enforce that every actions of a multinode use roles
e9fe783bf Show validation warnings when (re)submitting jobs in the web
ui
c3f8993fe Add NBD support to grub
07a693eaf device-types: minnowboard-common: permit to override
grub_autoboot_prompt
52c9f1ce8 device-types:: allow to change bootloader prompt at device
level
b3864a998 schema: accept zero MultiNode role count
78de67ce0 schema: accept test action parameters
5143b42f3 device-types: add sun5i-a13-olinuxino-micro
7403ddfd4 Remove all links to lavapdu and improve PDUDaemon docs
613038ccd doc: fix typo on --passwd argument
c56ab79e5 schema: add deploy.overlay module
5043f5e83 doc: make the message toward 3rd party packaging more
friendly
0140f42f3 Remove unsed argument
86a923cde Fix version/arch computation in docker debian-slim variants
79d239060 CI: use new sast template provided by gitlab
fce87ba18 cubietruck: use a longer bootloader prompt
83de037f0 Improve sql efficiency when generating lab report
4bab2f601 Improve "lava-server manage check"
714cdb587 Move back adduser call from postinst.py to
lava-server.install
668d132c6 device-types: base: detect TFTP ERROR: File not found
5bf642e3b scheduler: added AM6 device type
eb784fa69 Remove the top-level links from the docs
22f816bbb Add extra doc template code to add rel="canonical" links
95b43d160 boot minimal: allow to skip reseting the device
54f00eea7 Fix kexec boot failure
4819cb9db lava-test-raise: fix shell syntax
d718a4a9d device-type: add meson-g12a-x96-max
0f29a3e9e device-type: add meson-g12a-sei510
386c29ea0 device-type: add new meson-g12-common
affccf663 auto login: fix the prompts list after login
bcb243c65 debian: remove dependencies to unused jquery libs
eece2782b CI: skip debian specific tests if dpkg-query is not
installed
f2fdfa0f9 Improve error message
c974a8859 scheduler: rename rzn1 to rzn1d
b2cebf9a1 Only deploy the LAVA docs on a tag
a7a3f1f2f schema: allow namespace in user commands
f5b6edf73 Use enumerate whenever possible
f44347ce9 Remove "-master" (branch name) in docker tags
e09d68f19 Fix argument typo.
7cd503803 lava_scheduler_app: add barebox test
9ba1c02eb device-types: add imx6ul-pico-hobbit device
9e7a7ecd2 device-types: add base-barebox for typical barebox setup
6fda3991d dispatcher: add barebox boot action
1a6bca180 REST API: translate dictionary values to human readable form
e3b6fde17 Remove /etc/lava-dispatcher/lava-dispatcher.conf
2b0c7569e Upgrade bootstrap to v3.4.1
4cf162244 Use bootstrap minified css file
10396e6a0 release: add the script to push docker containers to docker
hub
2a7e17ab6 check: fix the check for systemd existence
bf4dc683e check: fix a crash when the uid is unknown
443a8b87b Add validate helper to manage jobs support
58eb62334 Remove unused imports
f51d9d000 rest-api: also list workers
2e7cae991 Add --host argument to pg_isready command call
b5090e4d9 debian/lava-dispatcher.dirs: remove usr/sbin
936541ff6 Fix typo in lava-modules.conf install path
Steve, for the LAVA team
--
Steve McIntyre steve.mcintyre(a)linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
Hello,
following the release of LAVA 2019.03, we released lavacli v0.9.5
This release brings one new command: "lavacli jobs validate"
This command send the given job definition to LAVA (version >= 2019.03).
LAVA will check the definition against the schema and report any errors.
As usual, lavacli could be installed from:
* pypi (https://pypi.org/project/lavacli/)
* debian (https://tracker.debian.org/pkg/lavacli) (coming soon).
Cheers
--
Rémi Duraffort
LAVA Team, Linaro
The 2019.03 tag has been pushed to master on git.lavasoftware.org.
Packages have been built in GitLab CI and are available at
apt.lavasoftware.org/release. Docker images for amd64 and arm64 have
been built in GitLab CI and are available from hub.lavasoftware.org
and https://hub.docker.com/u/lavasoftware .
Changes to LAVA UI configuration
================================
https://master.lavasoftware.org/static/docs/v2/advanced-installation.html#c…
Initial settings for a LAVA instance change over time as the
requirements change and dependencies improve internal security
implementations. Most instances will need some adjustment to the
apache configuration for the main LAVA UI in
/etc/apache2/sites-available/lava-server.conf and the LAVA packages do
not attempt to update this file automatically once admins have made
changes. Admins should therefore subscribe to the lava-announce
mailing list and make changes using separate configuration management.
apache2
=======
Browser caching can be improved by enabling mod_header in Apache to
allow LAVA to send appropriate cache control headers as well as mod
proxy and mod proxy_http:
$ sudo a2enmod header
$ sudo a2enmod expires
$ sudo service apache2 restart
LAVA server settings changes
============================
Django requires the allowed hosts to be explicitly set in the LAVA
settings, as a list of hostnames or IP addresses which LAVA is
allowed/expected to use. If this is wrongly configured, the UI will
raise a HTTP 500 error and you will get information in the output of:
lava-server manage check --deploy
or in
/var/log/lava-server/django.log
For example, /etc/lava-server/settings.conf for
https://lava.codehelp.co.uk/contains:
"ALLOWED_HOSTS": ["lava.codehelp.co.uk"],
See also
https://docs.djangoproject.com/en/1.11/ref/settings/#allowed-hosts
It is also important to enable ProxyPreserveHost in
/etc/apache2/sites-available/lava-server.conf
ProxyPreserveHost On
In some situations, you may also need to set USE_X_FORWARDED_HOST to
False in /etc/lava-server/settings.conf
See also
https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-USE_X_FORW…https://git.lavasoftware.org/lava/lava/merge_requests/400
python3-junit.xml
=================
python3-junit.xml >= 1.8 is required for REST API calls when running A
suitable package has now been backported to provide this functionality
for Stretch.
https://packages.debian.org/stretch-backports/python3-junit.xml
Note: there is a very similarly named package which will **not** work
(python3-junitxml). Look for the dot in the package name of the
correct package: python3-junit.xml
When installing, remember to allow stretch-backports:
$ sudo apt -t stretch-backports install python3-junit.xml
https://git.lavasoftware.org/lava/lava/merge_requests/357
Bump version of django-tables2
==============================
With 2019.01 and later, django-tables2 version 1.14 does not pick up
the customisation of the tables, so orderable icons and search forms
and pagination are missing.
It was not practical to backport django-tables2 to stretch-backports
due to complexities in the build dependencies of the package. So this
update has been made available via apt.lavasoftware.org
Stretch users will find a new version of python3-django-tables2 will be
installed alongside 2019.03.
https://git.lavasoftware.org/lava/lava/merge_requests/390
Prevent action.timeout exceeding job.timeout
============================================
This change affects the test jobs you submit to LAVA. The test job will
warn if any one action timeout exceeds the overall job timeout. LAVA is
adopting a stricter submission schema and this is the first common issue.
If you have a repository containing a number of test jobs, you can
check all of the jobs in one go by passing the path to a script in the
lava-common package.
$ sudo apt install lava-common
$ /usr/share/lava-common/lava-schema.py jobs ./*.yaml
You may want to edit the timeouts when resubmitting older test jobs.
When running /usr/share/lava-common/lava-schema.py, there may be a
range of other errors reported too. In future releases, LAVA will no
longer silently ignore unknown or obsolete YAML blocks. If there is
an error in the submission, the expanded schema will report it.
It is worthwhile running /usr/share/lava-common/lava-schema.py
against the output of all your test job submission templates. If you
use templating and CI, then you could add the check to the CI of each
change to the templating. Just ensure that the lava-common package is
upgraded to the same version as the instance to which the test jobs
would be submitted.
In a future release, this expanded schema check will be included into
lavacli, lava-server manage and as an XMLRPC call which can be used
before the submission. For now, only the action.timeout exceeding
job.timeout warning is being enabled in 2019.03.
https://git.lavasoftware.org/lava/lava/merge_requests/413
Changes in this release
=======================
60b9c653f checks: fix crash if the cmdline is not in the expected format
86cfe1ac2 Update reprepro-release.sh
6129fe7c0 tweak reprepro-release for new directory
b0cf8d779 rzn1: allow to boot using a fit image available over tftp
3e4faf6e1 Add qcs404-evb-1k-specific health check and fix a typo
9fd571b3e Fix crash in "check --deploy"
07a89fab3 device-type: Add Qualcomm's qcs404-1k and qcs404-4k
dcf3f2c6c debian: lava meta pkg: allow to use chrony instead of ntp
2d8e09126 (codehelp/master) schema: improve schema validator
8fa55d888 xmlrpc: allow to validate job definition schema
5aa407f04 Add default volumes for lava-slave-docker types
8f2a8cbe6 schema: move validate function to lava_common.schema
0b38b5805 Update copyright years across the LAVA source tree
5c17463e1 Package lava-schema.py in lava-common
a846ad088 Provide symlink for favicon
785906e1b Make timeout check non-fatal
6373b6869 lava-schema: allow to read from stdin
25812ecc7 schema: allow for any objects for test def params
861ec2fb3 Revert change to USE_X_FORWARDED_HOST
f26f61ee7 Avoid spurious boot command list warning
4915231e4 Skip analysis jobs on tags
d53814cc0 Add docs on advanced Apache and container usage
b42520584 Mention lava-lxc-mocker in docs
93828f63d device-types: add Intel upsquare
170dc7547 device-types: add pine64+
f1bbb9bb8 lava-server-gunicorn: allow to set the bind address
354077cb4 Schema: handle recovery deploy and boot methods
ae39f9004 Schema: add deploy to "download" method
7b9119bd7 CI check more sample jobs
168c4db20 Bump version of django-tables2 backport
1b0749092 Fix more sample jobs and ad schroot boot
e30c2f03d Fix typo
ea2987a08 Fix error in maintenance breadcrumb
291aa4ddb Make kernel messages wait sensitive to force_prompt
f3adeb798 Add cls.LT and cls.EXACT to models.DateTimeField
84ad3683a Check health-check YAML and schema
9299d5263 REST API: test /junit
1d5ea3cf7 Require python3-junitxml >= 1.8 for REST API calls
dc5a6fe48 templates: handle multinode vs singlenode
5f01a79c6 CI: run pylint to find common errors
00db03351 Fix some crashes found by pylint3
5a92736ad CI: test templates against the schema
3b5c9d5ef Add extra U-Boot bootloader error
2929ba8f3 Raise the codeclimate complexity threshold
d84756688 apache2: send the right cache-control headers when available
3cc42420f rzn1: do not set uboot_needs_interrupt explicitely
656666cb7 maintenance: fix crash after new state machine
00cea9df4 Device/Worker: remove deprecated method
576146730 Fix missing imports
32960bdd5 Handle duplicate listings in requires
19fdfd969 Move get_domain() to dbutils
01d354d1b img2simg and simg2img now available in a new package
bb62625f3 Fixup changes in the bandit overrides
ae25f1fe1 Update copyright for recent changes
4c33f7a54 Wait for the prompt when using shell on DUT
5662f734a code_quality: install radeo in the setup step
19bda4ac5 Support using radon to output code-climate data
489bc8416 schema: check namespace syntax
6406e1368 Improve inline test schema
0ae489101 Schema: check that jobs with secrets are not public
36d7f4ebb Check that timeouts are shorter than job timeout
fd225060e Improve schema validator
3a2fc459b Prevent action.timeout exceeding job.timeout
e6bfc3f43 Allow check --deploy inside Docker
69053fb82 Ignore DeprecatedWarnings on buster in CI
01fa79469 ci-run: use pytest-3 by default
075bf1f28 Fix crash when pre-test install failed
0c1372f9d Fix crash with older python3-tap versions
02097dd41 Fixup 4ebbc234 - put new test in correct location
feff6c914 CI do not run analyze for tags
9d4a91251 Fix documentation
2354f3899 doc: mention the minimal boot method
354f33ac2 imx8m device type restructuration.
4e1090982 base.jinja2: Allow interrupt-newline to be disabled in u-boot
958068fc6 Fix more sample job syntax
27fdc5107 device-types: add hsdk
dd4663b0a multinode: use character_delay
4ebbc234b power: use run_cmd instead of run_command run_cmd: accept
command as a string
9d9650939 Add missing pytest dependency for autopkgtest
d5bf2c248 Fix typo in f6bb9de2
b3eefdedb Fix missing change which breaks Debian CI testing
> Steve, for the LAVA team
--
Steve McIntyre steve.mcintyre(a)linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
