Hi folks,
The 2019.04 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are available at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and are
available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=======================
Device-types
============
New device-types
-----------------
New supported devices:
* A13-OLinuXino-MICRO (Allwinner)
* AM65x-GPEVM (aka am6) (TI)
* ar9331-dpt-module (DPTechnics)
* BeagleBone Black
* dove cubebox (Marvell)
* i.MX23 (Olimex)
* i.MX27-phytec-phycard-s-rdk (PHYTEC)
* i.MX28-duckbill (EnOcean)
* i.MX53-qsrb
* i.MX6ul-pico-hobbit
* i.MX6dl-riotboard (RIoTboard)
* meson-g12a-sei510 (Amlogic)
* meson-g12a-x96-max (Amlogic)
* SoCFPGA Cyclone V SoCrates (Altera)
cubietruck
----------
The bootloader prompt for `cubietruck` has been updated from `"sun7i"`
to `"sun7i# "`. In fact, the shorter form was also matching the dtb
filename, making some job unreliable.
qemu
----
If one or more test actions are included in a qemu job, LAVA will
create a disk image and will attach it to qemu.
Some emulated devices have no bus available for attaching this image
(like cubieboard). Some emulated devices have an available bus but
qemu is unable to attach to it due to the selected architecture (like
vexpress).
For these boards, you need to set `guestfs_interface` to `None` and
add a device with `drive=lavatest` in the job `context`:
context:
extra_options:
- -device virtio-bk-device,drive=lavatest
guestfs_interface: None
rzn1
----
`rz1` has been renamed to `rzn1d` to match OpenEmbedded board name.
Boot methods
============
Barebox
-------
A new boot method called `barebox` has been contributed. This method,
allows to boot boards using the `barebox` bootloader. Currently, the
resources are deployed using `tftp`.
A brebox job would look like:
actions:
- deploy:
to: tftp
kernel:
url: http://.../zImage
nfsrootfs:
url: http://.../jessie-armhf-nfs.tar.gz
compression: gz
dtb:
url: http://.../am335x-bone.dtb
…
- boot:
method: barebox
commands: nfs
prompts:
- 'root@jessie:'
…
The following device-types support barebox:
* ar9331-dpt-module (DPTechnics)
* BeagleBone Black
* dove cubebox (Marvell)
* i.MX23 (Olimex)
* i.MX27-phytec-phycard-s-rdk (PHYTEC)
* i.MX28-duckbill (EnOcean)
* i.MX53-qsbr
* i.MX6dl-riotboard (RIoTboard)
* SoCFPGA Cyclone V SoCrates (Altera)
dfu
---
Allow to use u-boot based dfu instead of the hardware one
On some boards, the hardware dfu serial number is set to 0000000. If
the board provides two boot media (NOR and NAND for example), we can
have a recovery u-boot on one media that will be used to flash the
second media using dfu.
actions:
- deploy:
to: tmpfs
images:
sf_uboot0:
image_arg: --alt sf_uboot0 --download {sf_uboot0}
url: http://.../u-boot-ddr.itb
- boot:
namespace: dfu
method: dfu
Grub
----
When booting with grub, users can now ask to serve the filesystem
from NBD instead of ramdisk or NFS.
- boot:
method: grub
commands: nbd
Minimal
-------
By default LAVA will reset the board power when executing the minimal
boot action. Users can skip this step by adding ``reset: false``. This
can be useful when testing bootloader in interactive tests and then
booting to the OS.
- boot:
method: minimal
reset: false
prompts:
- 'root@debian:~#'
- '/ #'
kexec
-----
The `kexec` boot method has been fixed. Thanks to this support, users
can test this kernel functionality:
- boot:
method: kexec
kernel: /boot/vmlinuz-3.16.0-4-amd64
initrd: /boot/initrd.img-3.16.0-4-amd64
command: /sbin/kexec
kernel-config: /boot/config-3.16.0-4-amd64
boot_message: Starting new kernel
options:
- "--reuse-cmdline"
auto_login:
login_prompt: "login:"
username: root
REST API
========
A new endpoint has been added to list `workers` at
`/api/v0.1/workers`.
Some fields are now using human readable names instead of opaque
constants:
* device:
* health
* state
* device-type:
* health_denominator
* TestJob:
* health
* state
* visibility
* submitter
Job definition schema
=====================
Web UI
------
When submitting or resubmitting a job from the web interface, the new
schema validator will be used. Any errors found by the new schema
validator will be reported as a warning.
Admin helper
------------
A new helper has been created for admins to validate old job
definitions on their local instance:
lava-server manage jobs validate --newer-than 1d
lava-server manage jobs validate --newer-than 1d --submitter lava-health
Job context
-----------
The schema validator is now checking the content of the `context`
dictionary. Only the following keys are now allowed:
* `arch`, `boot_console`, `boot_root`, `cpu`, `extra_options`, `guestfs_driveid`, `guestfs_interface`, `guestfs_size`, `machine`, `memory`, `model`, `monitor`, `netdevice`, `serial`, `vga`
* `bootloader_prompt`, `console_device`, `extra_kernel_args`, `extra_nfsroot_args`, `kernel_loglevel`, `kernel_start_message`, `lava_test_results_dir`, `menu_interrupt_prompt`, `mustang_menu_list`, `test_character_delay`, `tftp_mac_address`
Jobs using keys that are not listed in this list will be rejected.
If you think that some more keys should be whitelisted, please send a
patch.
Docker
======
Debian slim
-----------
The `lava-server` and `lava_dispatcher` docker images are now based on
`debian:stretch-slim`. This will reduce the size of the images
slightly.
lavaserver uid
--------------
In order to prevent any future issues, the `uid` and `gid` of the
`lavaserver` user has been fixed to `200`.
If you are upgrading from a previous lava-server docker image and
keeping the data (job logs, device dictionaries, ...) you will need to
change the owner of the files and directory:
chown -R lavaserver:lavaserver /etc/lava-server/dispatcher-config
chown -R lavaserver:lavaserver /var/lib/lava-server/
Full list of changes
====================
ec8e42c5f schema: vadidate context dictionnary
2e6964c17 device-types: fix board name imx53-qsbr -> imx53-qsrb
52e55a805 Tweak the reprepro-release.sh script again
73ad6ef10 Update the home page contents
d929cc8f9 Upgrade jQuery to v3.4.0 after the last security issue
8dd24244e Revert "device-types: base: detect TFTP ERROR: File not
found"
215a9ca04 Revert "device-types: use array instead of sting
concatenation"
976541d3a Update docs, adding more references to apt.lavasoftware.org
d0e9d7279 GitHelper: when failing, print git command stdout/stderr
e5671dbf1 u-boot-dfu tests: skip if dfu-util is not available
0b660a124 lava shell helpers: fix shellcheck warnings
7f02689c2 Allow to use u-boot based dfu instead of the hardware one
f81e83f31 qemu: Add an ID to the qemu storage
e063102e2 Convert offset and limit query parameters to ints
6363d1020 results: use yaml CDumper instead of the python dumper
d259def87 Fix documentation about template caching
2dbb0e059 device-types: use array instead of sting concatenation
0b3f671bf Fix crash when validating an invalid multinode job
0b6550d6f sample jobs: improve syntax
7c38216ea Schema: remove "live" argument
7e0a8e415 Fix file permissions
562be7cc6 Remove unused allow_modify from grub-nbd sample job
6f25a8086 device-types: add socfpga-cyclone5-socrates device
660b5b126 device-types: add ar9331-dpt-module device
81fa52c07 device-types: add dove-cubox device
65dacd712 device-types: add imx6dl-riotboard device
4566a4639 device-types: add imx53-qsbr device
7b1d2d0b9 device-types: add imx27-phytec-phycard-s-rdk device
2735228e1 device-types: add imx28-duckbill device
91abf4c2c device-types: add imx23-olinuxino device
efec57256 sample_jobs: add barebox job definitions
7b8e78b3c device-types: add parameters section to barebox type
b8102f433 device-types: add helper for barebox test and use on
imx6ul-pico-hobbit
ca800a7b6 xmlrpc: do not raise if jobs.list is empty
705d0aeeb schema: improve inline test definition schema
f5e06bf7d lava-schema: print yaml syntax errors
000420042 schema: allow for auto_login in fastboot and kexec
29708cbb0 schema: allow for no compression
883e24be4 schema: enforce that every actions of a multinode use roles
e9fe783bf Show validation warnings when (re)submitting jobs in the web
ui
c3f8993fe Add NBD support to grub
07a693eaf device-types: minnowboard-common: permit to override
grub_autoboot_prompt
52c9f1ce8 device-types:: allow to change bootloader prompt at device
level
b3864a998 schema: accept zero MultiNode role count
78de67ce0 schema: accept test action parameters
5143b42f3 device-types: add sun5i-a13-olinuxino-micro
7403ddfd4 Remove all links to lavapdu and improve PDUDaemon docs
613038ccd doc: fix typo on --passwd argument
c56ab79e5 schema: add deploy.overlay module
5043f5e83 doc: make the message toward 3rd party packaging more
friendly
0140f42f3 Remove unsed argument
86a923cde Fix version/arch computation in docker debian-slim variants
79d239060 CI: use new sast template provided by gitlab
fce87ba18 cubietruck: use a longer bootloader prompt
83de037f0 Improve sql efficiency when generating lab report
4bab2f601 Improve "lava-server manage check"
714cdb587 Move back adduser call from postinst.py to
lava-server.install
668d132c6 device-types: base: detect TFTP ERROR: File not found
5bf642e3b scheduler: added AM6 device type
eb784fa69 Remove the top-level links from the docs
22f816bbb Add extra doc template code to add rel="canonical" links
95b43d160 boot minimal: allow to skip reseting the device
54f00eea7 Fix kexec boot failure
4819cb9db lava-test-raise: fix shell syntax
d718a4a9d device-type: add meson-g12a-x96-max
0f29a3e9e device-type: add meson-g12a-sei510
386c29ea0 device-type: add new meson-g12-common
affccf663 auto login: fix the prompts list after login
bcb243c65 debian: remove dependencies to unused jquery libs
eece2782b CI: skip debian specific tests if dpkg-query is not
installed
f2fdfa0f9 Improve error message
c974a8859 scheduler: rename rzn1 to rzn1d
b2cebf9a1 Only deploy the LAVA docs on a tag
a7a3f1f2f schema: allow namespace in user commands
f5b6edf73 Use enumerate whenever possible
f44347ce9 Remove "-master" (branch name) in docker tags
e09d68f19 Fix argument typo.
7cd503803 lava_scheduler_app: add barebox test
9ba1c02eb device-types: add imx6ul-pico-hobbit device
9e7a7ecd2 device-types: add base-barebox for typical barebox setup
6fda3991d dispatcher: add barebox boot action
1a6bca180 REST API: translate dictionary values to human readable form
e3b6fde17 Remove /etc/lava-dispatcher/lava-dispatcher.conf
2b0c7569e Upgrade bootstrap to v3.4.1
4cf162244 Use bootstrap minified css file
10396e6a0 release: add the script to push docker containers to docker
hub
2a7e17ab6 check: fix the check for systemd existence
bf4dc683e check: fix a crash when the uid is unknown
443a8b87b Add validate helper to manage jobs support
58eb62334 Remove unused imports
f51d9d000 rest-api: also list workers
2e7cae991 Add --host argument to pg_isready command call
b5090e4d9 debian/lava-dispatcher.dirs: remove usr/sbin
936541ff6 Fix typo in lava-modules.conf install path
Steve, for the LAVA team
--
Steve McIntyre steve.mcintyre(a)linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
Hello,
following the release of LAVA 2019.03, we released lavacli v0.9.5
This release brings one new command: "lavacli jobs validate"
This command send the given job definition to LAVA (version >= 2019.03).
LAVA will check the definition against the schema and report any errors.
As usual, lavacli could be installed from:
* pypi (https://pypi.org/project/lavacli/)
* debian (https://tracker.debian.org/pkg/lavacli) (coming soon).
Cheers
--
Rémi Duraffort
LAVA Team, Linaro
The 2019.03 tag has been pushed to master on git.lavasoftware.org.
Packages have been built in GitLab CI and are available at
apt.lavasoftware.org/release. Docker images for amd64 and arm64 have
been built in GitLab CI and are available from hub.lavasoftware.org
and https://hub.docker.com/u/lavasoftware .
Changes to LAVA UI configuration
================================
https://master.lavasoftware.org/static/docs/v2/advanced-installation.html#c…
Initial settings for a LAVA instance change over time as the
requirements change and dependencies improve internal security
implementations. Most instances will need some adjustment to the
apache configuration for the main LAVA UI in
/etc/apache2/sites-available/lava-server.conf and the LAVA packages do
not attempt to update this file automatically once admins have made
changes. Admins should therefore subscribe to the lava-announce
mailing list and make changes using separate configuration management.
apache2
=======
Browser caching can be improved by enabling mod_header in Apache to
allow LAVA to send appropriate cache control headers as well as mod
proxy and mod proxy_http:
$ sudo a2enmod header
$ sudo a2enmod expires
$ sudo service apache2 restart
LAVA server settings changes
============================
Django requires the allowed hosts to be explicitly set in the LAVA
settings, as a list of hostnames or IP addresses which LAVA is
allowed/expected to use. If this is wrongly configured, the UI will
raise a HTTP 500 error and you will get information in the output of:
lava-server manage check --deploy
or in
/var/log/lava-server/django.log
For example, /etc/lava-server/settings.conf for
https://lava.codehelp.co.uk/contains:
"ALLOWED_HOSTS": ["lava.codehelp.co.uk"],
See also
https://docs.djangoproject.com/en/1.11/ref/settings/#allowed-hosts
It is also important to enable ProxyPreserveHost in
/etc/apache2/sites-available/lava-server.conf
ProxyPreserveHost On
In some situations, you may also need to set USE_X_FORWARDED_HOST to
False in /etc/lava-server/settings.conf
See also
https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-USE_X_FORW…https://git.lavasoftware.org/lava/lava/merge_requests/400
python3-junit.xml
=================
python3-junit.xml >= 1.8 is required for REST API calls when running A
suitable package has now been backported to provide this functionality
for Stretch.
https://packages.debian.org/stretch-backports/python3-junit.xml
Note: there is a very similarly named package which will **not** work
(python3-junitxml). Look for the dot in the package name of the
correct package: python3-junit.xml
When installing, remember to allow stretch-backports:
$ sudo apt -t stretch-backports install python3-junit.xml
https://git.lavasoftware.org/lava/lava/merge_requests/357
Bump version of django-tables2
==============================
With 2019.01 and later, django-tables2 version 1.14 does not pick up
the customisation of the tables, so orderable icons and search forms
and pagination are missing.
It was not practical to backport django-tables2 to stretch-backports
due to complexities in the build dependencies of the package. So this
update has been made available via apt.lavasoftware.org
Stretch users will find a new version of python3-django-tables2 will be
installed alongside 2019.03.
https://git.lavasoftware.org/lava/lava/merge_requests/390
Prevent action.timeout exceeding job.timeout
============================================
This change affects the test jobs you submit to LAVA. The test job will
warn if any one action timeout exceeds the overall job timeout. LAVA is
adopting a stricter submission schema and this is the first common issue.
If you have a repository containing a number of test jobs, you can
check all of the jobs in one go by passing the path to a script in the
lava-common package.
$ sudo apt install lava-common
$ /usr/share/lava-common/lava-schema.py jobs ./*.yaml
You may want to edit the timeouts when resubmitting older test jobs.
When running /usr/share/lava-common/lava-schema.py, there may be a
range of other errors reported too. In future releases, LAVA will no
longer silently ignore unknown or obsolete YAML blocks. If there is
an error in the submission, the expanded schema will report it.
It is worthwhile running /usr/share/lava-common/lava-schema.py
against the output of all your test job submission templates. If you
use templating and CI, then you could add the check to the CI of each
change to the templating. Just ensure that the lava-common package is
upgraded to the same version as the instance to which the test jobs
would be submitted.
In a future release, this expanded schema check will be included into
lavacli, lava-server manage and as an XMLRPC call which can be used
before the submission. For now, only the action.timeout exceeding
job.timeout warning is being enabled in 2019.03.
https://git.lavasoftware.org/lava/lava/merge_requests/413
Changes in this release
=======================
60b9c653f checks: fix crash if the cmdline is not in the expected format
86cfe1ac2 Update reprepro-release.sh
6129fe7c0 tweak reprepro-release for new directory
b0cf8d779 rzn1: allow to boot using a fit image available over tftp
3e4faf6e1 Add qcs404-evb-1k-specific health check and fix a typo
9fd571b3e Fix crash in "check --deploy"
07a89fab3 device-type: Add Qualcomm's qcs404-1k and qcs404-4k
dcf3f2c6c debian: lava meta pkg: allow to use chrony instead of ntp
2d8e09126 (codehelp/master) schema: improve schema validator
8fa55d888 xmlrpc: allow to validate job definition schema
5aa407f04 Add default volumes for lava-slave-docker types
8f2a8cbe6 schema: move validate function to lava_common.schema
0b38b5805 Update copyright years across the LAVA source tree
5c17463e1 Package lava-schema.py in lava-common
a846ad088 Provide symlink for favicon
785906e1b Make timeout check non-fatal
6373b6869 lava-schema: allow to read from stdin
25812ecc7 schema: allow for any objects for test def params
861ec2fb3 Revert change to USE_X_FORWARDED_HOST
f26f61ee7 Avoid spurious boot command list warning
4915231e4 Skip analysis jobs on tags
d53814cc0 Add docs on advanced Apache and container usage
b42520584 Mention lava-lxc-mocker in docs
93828f63d device-types: add Intel upsquare
170dc7547 device-types: add pine64+
f1bbb9bb8 lava-server-gunicorn: allow to set the bind address
354077cb4 Schema: handle recovery deploy and boot methods
ae39f9004 Schema: add deploy to "download" method
7b9119bd7 CI check more sample jobs
168c4db20 Bump version of django-tables2 backport
1b0749092 Fix more sample jobs and ad schroot boot
e30c2f03d Fix typo
ea2987a08 Fix error in maintenance breadcrumb
291aa4ddb Make kernel messages wait sensitive to force_prompt
f3adeb798 Add cls.LT and cls.EXACT to models.DateTimeField
84ad3683a Check health-check YAML and schema
9299d5263 REST API: test /junit
1d5ea3cf7 Require python3-junitxml >= 1.8 for REST API calls
dc5a6fe48 templates: handle multinode vs singlenode
5f01a79c6 CI: run pylint to find common errors
00db03351 Fix some crashes found by pylint3
5a92736ad CI: test templates against the schema
3b5c9d5ef Add extra U-Boot bootloader error
2929ba8f3 Raise the codeclimate complexity threshold
d84756688 apache2: send the right cache-control headers when available
3cc42420f rzn1: do not set uboot_needs_interrupt explicitely
656666cb7 maintenance: fix crash after new state machine
00cea9df4 Device/Worker: remove deprecated method
576146730 Fix missing imports
32960bdd5 Handle duplicate listings in requires
19fdfd969 Move get_domain() to dbutils
01d354d1b img2simg and simg2img now available in a new package
bb62625f3 Fixup changes in the bandit overrides
ae25f1fe1 Update copyright for recent changes
4c33f7a54 Wait for the prompt when using shell on DUT
5662f734a code_quality: install radeo in the setup step
19bda4ac5 Support using radon to output code-climate data
489bc8416 schema: check namespace syntax
6406e1368 Improve inline test schema
0ae489101 Schema: check that jobs with secrets are not public
36d7f4ebb Check that timeouts are shorter than job timeout
fd225060e Improve schema validator
3a2fc459b Prevent action.timeout exceeding job.timeout
e6bfc3f43 Allow check --deploy inside Docker
69053fb82 Ignore DeprecatedWarnings on buster in CI
01fa79469 ci-run: use pytest-3 by default
075bf1f28 Fix crash when pre-test install failed
0c1372f9d Fix crash with older python3-tap versions
02097dd41 Fixup 4ebbc234 - put new test in correct location
feff6c914 CI do not run analyze for tags
9d4a91251 Fix documentation
2354f3899 doc: mention the minimal boot method
354f33ac2 imx8m device type restructuration.
4e1090982 base.jinja2: Allow interrupt-newline to be disabled in u-boot
958068fc6 Fix more sample job syntax
27fdc5107 device-types: add hsdk
dd4663b0a multinode: use character_delay
4ebbc234b power: use run_cmd instead of run_command run_cmd: accept
command as a string
9d9650939 Add missing pytest dependency for autopkgtest
d5bf2c248 Fix typo in f6bb9de2
b3eefdedb Fix missing change which breaks Debian CI testing
> Steve, for the LAVA team
--
Steve McIntyre steve.mcintyre(a)linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
Changes to LAVA UI configuration
================================
https://master.lavasoftware.org/static/docs/v2/advanced-installation.html#c…
Initial settings for a LAVA instance change over time as the
requirements change and dependencies improve internal security
implementations. Most instances will need some adjustment to the
apache configuration for the main LAVA UI in
/etc/apache2/sites-available/lava-server.conf and the LAVA packages do
not attempt to update this file automatically once admins have made
changes. Admins should therefore subscribe to the lava-announce
mailing list and make changes using separate configuration management.
apache2
=======
Browser caching can be improved by enabling mod_header in Apache to
allow LAVA to send appropriate cache control headers as well as mod
proxy and mod proxy_http:
$ sudo a2enmod header
$ sudo a2enmod expires
$ sudo service apache2 restart
LAVA server settings changes
============================
Django requires the allowed hosts to be explicitly set in the LAVA
settings, as a list of hostnames or IP addresses which LAVA is
allowed/expected to use. If this is wrongly configured, the UI will
raise a HTTP 500 error and you will get information in the output of:
lava-server manage check --deploy
or in
/var/log/lava-server/django.log
For example, /etc/lava-server/settings.conf for
https://lava.codehelp.co.uk/ contains:
"ALLOWED_HOSTS": ["lava.codehelp.co.uk"],
See also
https://docs.djangoproject.com/en/1.11/ref/settings/#allowed-hosts
It is also important to enable ProxyPreserveHost in
/etc/apache2/sites-available/lava-server.conf
ProxyPreserveHost On
In some situations, you may also need to set USE_X_FORWARDED_HOST to
False in /etc/lava-server/settings.conf
See also
https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-USE_X_FORW…https://git.lavasoftware.org/lava/lava/merge_requests/400
python3-junit.xml
=================
python3-junit.xml >= 1.8 is required for REST API calls when running A
suitable package has now been backported to provide this functionality
for Stretch.
https://packages.debian.org/stretch-backports/python3-junit.xml
Note: there is a very similarly named package which will **not** work
(python3-junitxml). Look for the full stop in the package name of the
correct package: python3-junit.xml
When installing, remember to allow stretch-backports:
$ sudo apt -t stretch-backports install python3-junit.xml
https://git.lavasoftware.org/lava/lava/merge_requests/357
Bump version of django-tables2
==============================
With 2019.01 and later, django-tables2 version 1.14 does not pick up
the customisation of the tables, so orderable icons and search forms
and pagination are missing.
It was not practical to backport django-tables2 to stretch-backports
due to complexities in the build dependencies of the package. So this
update has been made available via apt.lavasoftware.org
Stretch users will find a new version of python3-django-tables2 will be
installed alongside 2019.02.
https://git.lavasoftware.org/lava/lava/merge_requests/390
Prevent action.timeout exceeding job.timeout
============================================
This change affects the test jobs you submit to LAVA. The test job will
warn if any one action timeout exceeds the overall job timeout. LAVA is
adopting a stricter submission schema and this is the first common issue.
If you have a repository containing a number of test jobs, you can
check all of the jobs in one go by passing the path to a script in the
lava-common package.
$ sudo apt install lava-common
$ /usr/share/lava-common/lava-schema.py jobs ./*.yaml
You may want to edit the timeouts when resubmitting older test jobs.
When running /usr/share/lava-common/lava-schema.py, there may be a
range of other errors reported too. In future releases, LAVA will no
longer silently ignore unknown or obsolete YAML blocks. If there is
an error in the submission, the expanded schema will report it.
It is worthwhile running /usr/share/lava-common/lava-schema.py
against the output of all your test job submission templates. If you
use templating and CI, then you could add the check to the CI of each
change to the templating. Just ensure that the lava-common package is
upgraded to the same version as the instance to which the test jobs
would be submitted.
In the next release, this expanded schema check will be included into
lavacli, lava-server manage and as an XMLRPC call which can be used
before the submission. For now, only the action.timeout exceeding
job.timeout warning is being enabled in 2019.02.
https://git.lavasoftware.org/lava/lava/merge_requests/413
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Hello,
following the release of LAVA 2019.01, we released lavacli v0.9.4
This release brings two new commands:
* "lavacli utils templates render" to render jinja2 templates
This will help users/admins to debug jinja2 templates issues
* "lavacli jobs queue" to show the current queue for the requested
device-types
This is only available since LAVA 2019.01
As usual, lavacli could be installed from:
* pypi (https://pypi.org/project/lavacli/)
* debian (https://tracker.debian.org/pkg/lavacli)
Cheers
--
Rémi Duraffort
LAVA Team, Linaro
The 2019.01 tag has been pushed to master on git.lavasoftware.org.
Packages have been built in GitLab CI and are available at
apt.lavasoftware.org/release. Docker images for amd64 and arm64 have
been built in GitLab CI and are available from hub.lavasoftware.org
and https://hub.docker.com/u/lavasoftware
New bootloader testing and recovery support
==================================
The recovery mode support already implemented for the hikey 6220 has
been used to provide recovery mode operations for the X15 by Milosz
Wasilewski to support bootloader testing with automated recovery.
Dropped default patterns and fixup dicts
===============================
Omits the V1 compatibility constants for default pattern and fixup
dict. Patterns and fix up dictionaries are now only available in Lava
Test Definition 1.0 test shells if the test writer explicitly adds the
parse: block to the test definition. The documentation has been
updated.
Ensure lxc-templates is installed
=========================
To use lxc, the recommended lxc-templates package must be installed to
provide the template files. When installing lava-dispatcher in Docker,
it is now useful to specify the lava-lxc-mocker package as the updated
LXC packages will fail to install inside Docker.
Final black changes
================
All LAVA Software Python code in lava.git has now been formatted using
black and the GitLab CI ensures that this formatting is retained. If
you are developing code to be merged into lava.git, ensure that you
have black installed and run it as the final step before contributing
your changes upstream or the CI will fail.
REST API using django-restframework
=============================
Added REST API for LAVA, including ability to export results in JUnit and TAP13.
Non-POSIX test action
=================
Add a new test method for non-posix shells. The test writter is able
to launch any command on the DUT and is responsible for giving the
right set of patterns that match failure and success.
Add a new boot method called "bootloader"
=================================
This method allows to boot the DUT, interrupt the bootloader and jump
into the bootloader shell.
Changes in this release
==================
ec7875b05 Enhance the Docker admin documentation
ffe2c0751 Build architecture aware docker images
e0c651aae Update docs for development changes
c0eeb4b28 schema: allow for namespaces with interactive tests
aa94f3ac4 Allow to set uboot_needs_interrupt in the device dict
79fb10a12 Fix debian installer deploy method
8b6b53be0 Allow to set fastboot deploy uboot commands
3975eb4f4 doc: add documentation about x15-bl device-type
b87dc6816 scheduler: add x15-bl device type
5d8227c91 scheduler: add recovery support for base-uboot.jinja2
4ddd44ad6 Tweak the development process doc
2bbf5c6e5 Update git repository links
1a9f51494 Drop bashism in vland helper scripts
7ebb3feab Retrieve the vland version in the protocol
f2f933399 Improve docs on health check setup actions
1e596b6ff Improve server|logs|publisher command line parser
68dd8ff79 device-types: allow to set boot_character_delay in device dict
92fa46213 scheduler: randomise the device list
c0c3eb0ef lava-modules.conf should be in /etc/modprobe.d/
2412366c9 Fix sample jobs timeouts
68edafac9 CI: install minicom for mps2plus tests
f3cbdac58 Add lava-server manage command for site
cc16e4b38 mps2: fix validation and documentation
32d18fa3b device-types: add rzn1 template
7a992201c device-types: qemu: Add support for MIPS/PPC/PPC64
e114b354e qemu: Remove root_partition from sample jobs
407d857f9 Fix sample job syntax
049eb5afc sample jobs: move kernel type to the deploy action
25f37dc62 vemsd: recovery_image is mandatory
596aa3b29 Enforce realpath for which check
68b8e54b1 doc: add information about SQUAD to custom-results-handling
1282c4a25 Remove leftover from py2 to py3 conversion
dd76f0797 Add voluptuous to lava-common dependencies
8041e1012 Rework interactive test
f6bb9de2f Improve job page rendering with large jobs
e9fa46977 Improve schema validator and move to lava_common
f116ef3af Add check-devices.py to lava-dev
02e4d9e88 Fix sample jobs syntax
b4d84f82f Drop default patterns and fixup dicts
9fbcca0e4 Separate device and device-type as distinct types
fb4d5785e Action: name is a class variable
e990310e0 Remove deprecated functions since django 1.9
eb25d433e Reorder middlewares according to django default config
ebf17cbe6 Remove deprecated module
8a317b0b2 Record JobError in validate()
3bc657e7b Fix sample jobs syntax
b7eb162a1 Ensure lxc-templates is installed
a3524ce9a Expand minimal configuration for connections
ef2fa8594 Results table: decrease the number of sql queries
4df8e8464 Handle downloads using "Content-Encoding: gzip"
f37c03159 Move sample job to the right directory
37a2fe7fa Remove "connection" from job definition
d70e2623c Add checks for the connection type
5260ae1b9 doc: fix broken links
33918bcb4 Update ./ci-run for lava_common/tests change
5af8543b1 Simplify docker boot command line build
f6be3692c Update emphasis lines of literalinclude blocks
da1d472b5 Fix problems with the purge maintainer script
28afc08eb Remove add-header mentions
832fb942b Use render() shortcut
7ef4ac466 Fix bandit warnings after black formatting
d483d94a5 Make flasher deploy retriable
7f3e23ee1 Do not add yaml_line to the job definition
c76620a55 sample jobs: remove mkimage_arch
5be4be588 Move test_info to the job level
9eb721166 'nfs_url' option does not exist anymore
7c347e04e Fix the increased SAST count from formatting changes
017285ce5 Remove outdated README file
b471ee62f Unify test directory name
46f298251 fix prospector warnings
f7973476b Final black changes for lava_dispatcher
6cae9e765 CI: fix black check
816a47b9b Fix typo in check-devices error message
bc844be5c Apply black to lava_dispatcher
40e6a25f8 Apply black to lava/
93a70fcfc Update restore instructions
4ef3b776a Test repeat fix
01432ba06 Convert test_repeat to pytest
f9e782cc8 Fix repeat logic
72d563661 Update advice on restoring backups
bee7cc6da Update documentation example to use correct syntax
71276203a Fix inline example test job indent error
3d8cd9e88 Advise on upgrading python dependencies
e22dbf0c3 Fix states in maintenance.py
2cf6b51ab interactive: allow to only wait, not sending any command
bfb7cb47c Fix test stages computation when mixing test types
8bdc42ef3 Make StdoutTestCase.pipeline_reference a class method
989b21688 Search the queue by requested_device_type
32a4c821b notification: protect against potential crashes
b7254aec6 XMLRPC-API: decrease the number of sql requests
e65cfdc09 device-types: qemu: Add support for SPARC/SPARC64/MIPS64
ade4b7e0c Apply black to lava_scheduler_app
95aa92f9f Apply black to doc/
54c18a449 Apply black to share/, setup.py and version.py
c74c15d06 Simplify black listing
fd04401a9 Apply black to lava_results_app
8f100d82b Apply black to lava_server
7aa96e077 Apply black to lava_common
5e6e902fa Update documentation for new test action support
bfa253f9b Document the docker deploy
84d202b28 Device table: tag with <i> when worker is undefined.
009e21b53 doc: explain how to serve LAVA under /lava
d86d457f7 device-types: add meson-gxl-s805x-p241
8fb830b3d REST API: allow to export results in JUnit and TAP13
b50800bfc Add a new test method for non-posix shells
ce2627669 Let udev_trigger use the network from udev
172c6f22c Add a new boot method called "bootloader"
4e7248342 Add a qemu-aarch64 device-type
c715f5252 lava_results_app: fix exception when no action_metadata
229242900 Fix deprecation warnings
6cc1c7966 Set ProxyPreserveHost on.
b801a30f4 Create an out-of-tree symlink for djangorestframework
e6678963c Fix rest_framework static symlink
cac2cbb7f Allow dependencies of reprepro-master on schedules
438e23cab Enable reprepro-master
9788877eb checks: log an error when settings.conf is invalid
ad6a2c61d CI: only run deploy jobs for scheduled jobs
f4530df6b TC2 device jinja changes
349ee8175 Remove unused command migrate-lava
6690af7cf CI: skip some tests when lxc-start is not installed
cd7d9e21b Fix crash if dpkg-query does not know about a file
b9cb118ef Add docs on vemsd and mps deployments
d0fd22894 Update the docker-admin page for new links
9bac98720 REST API reduce the number of SQL requests
4bfad2969 Copy lava-lxc-mocker in the publish stage
dacd2a41c Fix lava_rest_app support in ci-run
421e5283d CI: use new sast syntax
0d75252a4 Apply black to lava_server files
b28764d28 Fix unit test exception handling
8e38d322e Add REST API for LAVA
5a55d6646 Update the release repository only on a tag
b55b1c1ba Report the version of tools
3e8f0eb5e Replace with a more pythonic way.
cef780613 Fix doc typo causing build error.
62150f062 Apply black to all migrations
7d040ca09 Optimise to reduce the compare cost.
688df87de Fix: MPS Serial Buffer
8a3d6a587 monitor: fix accepts logic
ade6320a2 Replace the expired job link with a valid one.
bfa558f30 CI build aarch64 docker image using debian 10
ef1c54427 Make navbar and userlinks seperate templates
9270725c9 Correct the path of file "commands.py"
4e7d05696 Tidy up ci-run
73d3a74a4 Ensure sample_jobs validate against the schema
0be0739ab Extend timeouts of example test jobs
b024c2c69 Skip OverlayAction if the overlay is not needed
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Hi,
In founding the LAVA Software Community Project, the team planned to
open up LAVA development more. As already announced by Neil in
September, we have already moved our infrastructure to a GitLab
instance and LAVA developers and users can collaborate there. [1]
The next step in our process is to also open our regular development
design meetings to interested developers. The LAVA design meeting is
where the team gets together to work out deep technical issues, and to
agree on future development goals and ideas. We run these as a weekly
video conference using Google Hangouts Meet [2], We now wish to
welcome other interested developers to join us there too, to help us
develop LAVA.
Summaries of the meetings will be posted regularly to the lava-devel
mailing list [3], and we encourage interested people to subscribe and
discuss LAVA development there.
[1] https://git.lavasoftware.org/
[2] https://meet.google.com/qre-rgen-zwc
[3] https://lists.lavasoftware.org/mailman/listinfo/lava-devel
Cheers,
--
Steve McIntyre steve.mcintyre(a)linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
We've done a little bit of advance planning and there are now three
milestones which developers will be using to schedule upcoming
releases.
https://git.lavasoftware.org/groups/lava/-/milestones
Issues and merge requests have been assigned to the relevant milestone.
All dates are approximate.
We will also be announcing (here) when the next release would include
database migrations so that admins have more notice of scheduled
downtime. When a milestone includes database migrations, other changes
which also include a database migration will be moved to that
milestone so that admins can minimise total downtime.
The next release is expected to include the changes from the 2018.12 milestone.
The 2019.02 milestone already includes database migrations.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Changes in the master branch (and daily build packages) now check the
ALLOWED_HOSTS setting for each master in your instance(s) in
/etc/lava-server/settings.conf
Please check your /etc/lava-server/settings.conf support for
ALLOWED_HOSTS in Django as this will be required to upgrade to the
next release of LAVA Software.
Remember to always restart lava-server-gunicorn after any change to
/etc/lava-server/settings.conf
See also https://master.lavasoftware.org/static/docs/v2/pipeline-debug.html#check-la…
and https://master.lavasoftware.org/static/docs/v2/pipeline-debug.html#displayi…
(new help sections on using and checking LAVA setttings.
Django docs: https://docs.djangoproject.com/en/1.11/ref/settings/#allowed-hosts
"""
This is a security measure to prevent HTTP Host header attacks, which
are possible even under many seemingly-safe web server configurations.
...
Django also allows the fully qualified domain name (FQDN) of any
entries. Some browsers include a trailing dot in the Host header which
Django strips when performing host validation.
...
When DEBUG is True and ALLOWED_HOSTS is empty, the host is validated
against ['localhost', '127.0.0.1', '[::1]'].
"""
If this setting is wrong, updated LAVA packages will fail to serve
HTTP and the /var/log/lava-server/django.log file will record messages
like:
ERROR 2018-11-26 18:12:52,091 exception Invalid HTTP_HOST header:
'lava.codehelp.co.uk'. You may need to add
'lava.codehelp.co.uk' to ALLOWED_HOSTS.
(In this case, that was exactly the correct action)
"ALLOWED_HOSTS": ["lava.codehelp.co.uk"], to
/etc/lava-server/settings.conf and restarting lava-server-gunicorn
fixed lava.codehelp.co.uk
See also https://git.lavasoftware.org/lava/lava/issues/173
When checking this change, it is useful to also set DEBUG to true as
Django will then show you all of the headers and environment
variables. Do remember to turn DEBUG back to false because these
variables are a security risk in themselves.
Another setting which might be useful, depending on your configuration:
"USE_X_FORWARDED_HOST": true,
It is safe to make this change in advance of the upgrade of LAVA but
do test on an instance which is as close as possible to the network
configuration of your production instance(s).
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Main changes in 2018.11
===================
0eb0d3ed7 Set requirements for requests to avoid CVE - as announced
previously python3-requests needs to be updated to protect against a
CVE in that package.
If installing 2018.11 on a Stretch system, ensure you have updated
python3-requests to the version from stretch-backports.
f7bbf6a8 Visibility of worker status effects on device views-
(improved in subsequent commits in this release). Indicate the status
of the worker when displaying information about the device so that it
is clear why the device is still Idle if there is a problem with the
worker.
ac0097c4 Fix 500 when rendering device dict page - if the device
dictionary is invalid.
0f7a7ec8 Prevent index out of range error and 97132e91 Extend 0f7a7ec8
to other command_output comparisons and 06203dfe Add a run_cmd helper
to Action class - the Action run_command support had become complex
and problematic. A new parsed_command has been added and a simpler
run_cmd. In time, existing use of run_command will be phased out in
preference for one of the two new commands. This should make it easier
to spot why certain dispatcher commands failed. This was triggered by
intermittent problems running simg2img and img2simg for AOSP.
4736b01d How to test and recover bootloaders in LAVA -
https://master.lavasoftware.org/static/docs/v2/bootloaders.html
ff37c802 Add docs on criteria for LAVA on other distributions -
https://master.lavasoftware.org/static/docs/v2/debian.html
Issues closed in 2018.11
===================
https://git.lavasoftware.org/lava/lava/issues?scope=all&utf8=%E2%9C%93&stat…
Full changelog
===========
ff37c802c Add docs on criteria for LAVA on other distributions
4736b01dc How to test and recover bootloaders in LAVA
2cae2b413 Fix lava-master crash when device yaml is invalid
570fb324d Allow sdist to be passed to setup.py
51b3fe6f6 Fix missing part of aarch64 stretch deployment
f9a3671c1 docker: check method
563fac8da Enable aarch64/pkg-debian-9 in the build tasks
7e71a56ea flasher: fix substitution when cmd contain whitespaces
5bb71d4a1 Package the requires.py script
d77c8d28f Add CI package script for stretch on aarch64
24eface95 docker: use the new Action.run_cmd helper
ddd986e8e Extend the base poweroff timeout.
9d49da1d4 Remove build from .gitignore
24022feb4 Fix missing return value in run step
b5c51dfa8 Force Juno to use NFS vers=3 and extend power off timeout
7ddae776c run_cmd: make spaces explicit
87f1e01e1 Capture the simg2img and img2simg output and log it
f40f0f2c5 Ensure apt is updated before trying to install
06203dfe2 Add a run_cmd helper to Action class
97132e916 Extend 0f7a7ec8 to other command_output comparisons
f0ebf8025 Do not send "\n" twice
ac0097c42 Fix 500 when rendering device dict page
57d80c2e2 Add deployment builds to snapshot directories.
9a1a6bfb1 Apply black to more files in lava_scheduler_app
c6bfbd5f8 Apply black to lava_dispatcher unit tests
0b615023d version: allow user to specify the branch to use
f7bbf6a8b Visibility of worker status effects on device views
f2d9b658e Apply black to lava_scheduler_app/api and tests.
7980d84c3 docker tag: move branch name to suffix, not prefix
0f7a7ec8f Prevent index out of range error
984164f68 Fix omission in package deployment script
d4b942b71 CI use pytest-3 instead of py.test-3
d77c7080c Fix regression in fb4a88388
504115553 Update docs on local dev builds
fb4a88388 Standardise on dots in the version string
4a1910049 Use the rollback support in requires.py
cb86cc09b Improve error message after 4c71c2ebd1
15d7a905f Include package artifacts into repositories
4c71c2ebd Provide more information in bootloader errors
abff27c8b CI: skip deploy when lavafed is running latest version
f691bc059 CI fix typo for "only.refs"
024b27694 Fix errors in docker-admin page
a937a29e4 CI: add missing requirement to pytest
08ea617a4 vland: fix crash when finalizing
987280afd Fix errors in docker-admin page
89f96b1d9 CI fix deployment script
c5de132b1 Deploy lavafed-master when scheduled
08eab0bcb Fix issues with unit test calculations
7e8a6f444 CI use the new arch specific images
7d2956e43 Sort the --names output of requires.py
0eb0d3ed7 Set requirements for requests to avoid CVE
8cd80dcb2 Build packages and docker images for AArch64
1ab6aaa9c Allow unit tests to run without /sys/class/misc/kvm
83649d77b Port 0e598e63 to the xmlrcp api
6335f73d8 Prevent crash if environment requested for non-POSIX
14b347c51 lava_results_app: convert Decimal objects to string
storing as YAML
26bf0af63 lava_results_app: add test case for YAML Decimal
object conversion
1744ea74c Change ownership of health checks and device-types
32a2a6051 Ignore gitlab-ci files when creating a release tarball
bd224fcd1 Update publish for changes in pkg/docker
2e5e60c9f Drop obsolete gitreview file
2018.11-1 has been uploaded to Debian unstable. This version should
migrate into buster, at which point an upload to stretch-backports
will be made too.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
