Hello,
following the release of LAVA 2019.05, we released lavacli v0.9.6
This release brings some improvements:
* "jobs cancel" can now cancel multiple jobs
* "utils templates render" accepts a context and can trim the output
* add a timeout to "jobs wait" command
In order to use the new aliases introduced/fixed in LAVA 2019.05, lavacli
v0.9.6 is required.
As usual, lavacli could be installed from:
* pypi (https://pypi.org/project/lavacli/)
* debian (https://tracker.debian.org/pkg/lavacli) (coming soon).
Cheers
--
Rémi Duraffort
LAVA Team, Linaro
Hi folks,
The 2019.05 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are available at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
will be available shortly from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=======================
Device-types
============
Aliases
-------
The device-type aliases feature has been totally redesigned. You can
now define multiple aliases for a given device-type. In a job
definition, users can use either the device-type or any of its
corresponding aliases.
New device-types
----------------
New supported devices:
* da850-lcdk (davinci)
* SoCA9 (Schneider)
frdm-k64f
---------
It is now possible to override the setting `resets_after_flash` in the
device dictionary, to support newer firmwares.
meson-gl12
----------
Fix the dtb and ramdisk addresses:
* dtb: **0x01070000** (was **0x0107f000**)
* ramdisk: **0x08000000** (was **0x04000000**)
RPi
---
Move the ramdisk load address to **0x02200000** (was **0x02100000**)
to allow more space for the kernel image and avoid them overlapping.
rzn1d
-----
Allow to use zImage.
Database
========
Migrations
----------
This new version creates several database migrations to:
* remove `TestJob.admin_notifications` field
* improve `DeviceType` aliases
* replace some CharField types with TextField to remove unnecessary
size limitations
These may incur a small amount of downtime during the upgrade process
as database migrations happen. A goal for the LAVA development team is
to reduce the frequency of database migrations to reduce this cost as
much as possible. The next **planned** set of database migrations is
in the 2019.08 release, in August.
dashboard_app
-------------
The dashboard_app (holding v1 results) has been fully removed,
including:
* migrations
* database model
* xmlrpc calls
Docker
======
multiarch
---------
Due to some limitations in the docker registry embedded in GitLab, we
are no longer building and providing multiarch docker images for every
commit. Multiarch images will still be available on [docker
hub](https://hub.docker.com/u/lavasoftware/) for every release.
lava-dispatcher
---------------
The dispatcher docker image now includes `docker-ce-cli`. This will
allow lava to control docker containers from within a lava-dispatcher
container, helping admins to participate in lavafed.
Server performances
===================
Template caching
----------------
The `USE_TEMPLATE_CACHE` setting variable has been removed. On newer
Django versions, this option is no longer useful and might even cause
problems.
Compressing job logs
--------------------
Support has been added for compressing log files on the server, using
xz (lzma) compression. Logs will be stored uncompressed as jobs are
running, but they may later be compressed by admins to reduce the
amount of space needed.
The `lava-server manage jobs compress` command
includes options to allow for management of this. Admins can use this
in cronjobs to automatically compress job logs older than 30 days, or
similar (`lava-server manage jobs compress --older-than 30d --slow`).
LAVA will automatically decompress old compressed logs on the fly if
they are ever needed again in future.
Job definition schema
=====================
Admin helper
------------
The `lava-server manage jobs validate` command can now send a mail to
the admin when an invalid job is found.
To make use of this, admins can setup a regular cronjob that will call:
```shell
lava-server manage jobs validate --mail-admins
```
Job context
-----------
The schema validator is now checking the content of the `context`
dictionary for both single and multinode jobs.
Two new allowed keys have been added: `uboot_extra_error_message` and
`uboot_needs_interrupt`
Only the following keys are now allowed:
* `arch`, `boot_console`, `boot_root`, `cpu`, `extra_options`, `guestfs_driveid`, `guestfs_interface`, `guestfs_size`, `machine`, `memory`, `model`, `monitor`, `netdevice`, `serial`, `vga`
* `bootloader_prompt`, `console_device`, `extra_kernel_args`, `extra_nfsroot_args`, `kernel_loglevel`, `kernel_start_message`, `lava_test_results_dir`, `menu_interrupt_prompt`, `mustang_menu_list`, `test_character_delay`, `tftp_mac_address`, `uboot_extra_error_message`, `uboot_needs_interrupt`
Jobs using keys that are not listed in this list will be rejected.
If you think that some more keys should be whitelisted, talk to us.
Extending job context
---------------------
Admins can extend the white list by updating `EXTRA_CONTEXT_VARIABLES`
in the settings:
```json
"EXTRA_CONTEXT_VARIABLES": ["custom_variable1", "variable_2"]
```
Don't forget to restart `lava-server-gunicorn` if you make changes
here.
lava-slave
==========
Socks proxy
-----------
When connecting with `lava-master` and `lava-logs`, `lava-slave` can
now use a socks proxy if needed.
In order to use a socks proxy, you should either:
* start lava-slave with `--socks-proxy localhost:8081`
* edit `/etc/lava-dispatcher/lava-slave` and add
`SOCKS_PROXY="--socks-proxy localhost:8081"`
Custom CSS
==========
This adds the possibility to include a custom style sheet in a LAVA
instance for applying overall branding to the GUI.
Add the link to the CSS in the `settings.conf` and restart
`lava-server-gunicorn`:
```json
"BRANDING_CSS": "https://www.example.org/css/style.css",
```
Full list of changes since 2019.04
==================================
5e4a36956 jobs compress: create the output.yaml.size file
if needed.
412b73788 device-type: add SoCA9 from Schneider
aab54ce66 Revert "Auth refactoring part 1."
396c7342f logutils: test the module functions
15bb54214 logutils: only create the index when needed
ed60e521b Add a command line helper to compress logs
e19c9a2cf Read logs from either output.yaml or output.yaml.xz
41672feee job logs: replace TestJob.output_file by logutils
functions
c3bf98bd6 job: remove unused/uneeded job_file_present variable
c5c881347 Remove the archived_job_file function
1e32ed252 LAVA branding: add possibility to include custom CSS
417c8b14b release: add a script to remove old docker images
0e08fa08e Clean up obsolete conffiles causing piuparts failures
1e9778114 Present the codeclimate data in human readable format
2fa918740 schema: the job visibility is required.
2fbc72031 Order aliases by name when rendering
1a10f4903 scheduler.jobs.validate: fix return value
e7ce9d63c schema: allow admins to extend the white list of
context variable
a615696cb lava-schema: factorize and allow to check device config
0c68b065c schema: add a device schema into lava_common
1ac122efe Add fastboot support to the meson-g12a-sei510
56583c703 debian: remove unused or redondant files
9654c526a Remove permission object support from
GroupObjectPermission methods.
8d13f2def Change API to update single object permissions.
f365005b1 Small code review changes.
444404943 Add LAVAServerError exception tree.
7c533d5c4 Remove shortened codename support.
c991c3c5c Auth refactoring part 1.
c2acdbdd8 debian: remove unused py3dist overrides
dadce92a6 doc: fix sample job schema
c505f1322 schema: add missing uboot_needs_interrupt to the context
ccfbf2aed schema: fix boot.minimal by adding reset and auto_login
b17b24e94 schema: also validate the multinode context variable
0f13bc60a Change notification CharField types to TextField instead
4be183e2d unittests.sh was renamed to dispatcher-unittests.sh.
d6b642471 Partially revert 3d0ddb402 to fix some tables issues
ff613da21 device: improve the recent test jobs table
e21f2ddbb Use .exists() instead of len(.all())
bb3ca4525 XMLRPC API: fix crash in scheduler.aliases
85fc254b7 schema: fix remaining syntax error in the sample jobs
9cccb2e3a docs: Update docs for DeviceType alias
5b6424218 scheduler: add test for alias submissions
62c0e08ec scheduler/api: fix XML-RPC API for Aliases
ec74e9ed7 scheduler: avoid device type-alias name conflict
3d0ddb402 scheduler: modify alias to be 1-many relation
a6e9c6d2f lava-slave: add an option to use a socks proxy
7a18fc022 schema: allow uboot_extra_error_message in the context
711a20131 schema: fix definition of test definition option
"lava-signal"
b0088485c doc: fix emphasis
aefc87b26 Improve the README and use markdown for Gitlab rendering
a4b350516 fix pyocd boot action method to print out version of
binary used
c045401d1 Do not use minified version of JS and CSS files
55afae484 schema: add "new_connection" boot method
ed4eeda0a doc: add missing label
851b4bc33 version.py: fix git check when using worktrees
72b9dd18c Change the 'fastboot_sequence'
383525981 user commands: use run_cmd instead of run_command
b1c84a1d6 validate: allow to report invalid jobs to admins
82aa705b8 Fix multinode jobs validation
b10bc826f Use the static template function
d48ba02c6 Make USB/DFU/UDEV timeouts into Infra Errors
643180264 debian: list every minimun versions for buster
0dd5dc1d6 lava_common: remove unused dependency on pyzmq
f1c1372ca doc: fix syntax to match schema
759740e1c device-types: rpi: allow extra space for kernel image
3bd8f0d8a Fix file permissions
8139950b1 Remove USE_TEMPLATE_CACHE option
44c5e6ca5 scheduler: update rzn1d device-type to boot zImage
3fced42f5 Fix the dtb/ramdisk addresses
750d8f298 Document the "command" action block
0447e8327 doc: replace lava-tool by lavacli whenever possible
8c73a28ed Update packaging for dashboard removal
1c6be7f71 Remove leftover from v1 dashboard
cc7e86fa7 Remove dashboard_app
c4a4c3e3b device-type: add da850-lcdk
3a16ca954 Remove admin_notifications field
1a20620be CI: do not build anymore multiarch images
796c98eb3 CI: fix tag selection
cc2bc7509 add systemd support for lava-os-build
a7c483bdb Add overriding resets_after_flash to support newer
FRDM-K64F boards
537773c9e TestJob duration: return only seconds (and not
miliseconds)
5ae7f5a1a Add a helper script for Debian release
4b2c07da7 Fix broken seealso syntax in docs
447bc9344 doc/v2/timeouts: fix http-download action name typos
Steve, for the LAVA team
--
Steve McIntyre steve.mcintyre(a)linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
Hi folks,
The 2019.04 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are available at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and are
available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
=======================
Device-types
============
New device-types
-----------------
New supported devices:
* A13-OLinuXino-MICRO (Allwinner)
* AM65x-GPEVM (aka am6) (TI)
* ar9331-dpt-module (DPTechnics)
* BeagleBone Black
* dove cubebox (Marvell)
* i.MX23 (Olimex)
* i.MX27-phytec-phycard-s-rdk (PHYTEC)
* i.MX28-duckbill (EnOcean)
* i.MX53-qsrb
* i.MX6ul-pico-hobbit
* i.MX6dl-riotboard (RIoTboard)
* meson-g12a-sei510 (Amlogic)
* meson-g12a-x96-max (Amlogic)
* SoCFPGA Cyclone V SoCrates (Altera)
cubietruck
----------
The bootloader prompt for `cubietruck` has been updated from `"sun7i"`
to `"sun7i# "`. In fact, the shorter form was also matching the dtb
filename, making some job unreliable.
qemu
----
If one or more test actions are included in a qemu job, LAVA will
create a disk image and will attach it to qemu.
Some emulated devices have no bus available for attaching this image
(like cubieboard). Some emulated devices have an available bus but
qemu is unable to attach to it due to the selected architecture (like
vexpress).
For these boards, you need to set `guestfs_interface` to `None` and
add a device with `drive=lavatest` in the job `context`:
context:
extra_options:
- -device virtio-bk-device,drive=lavatest
guestfs_interface: None
rzn1
----
`rz1` has been renamed to `rzn1d` to match OpenEmbedded board name.
Boot methods
============
Barebox
-------
A new boot method called `barebox` has been contributed. This method,
allows to boot boards using the `barebox` bootloader. Currently, the
resources are deployed using `tftp`.
A brebox job would look like:
actions:
- deploy:
to: tftp
kernel:
url: http://.../zImage
nfsrootfs:
url: http://.../jessie-armhf-nfs.tar.gz
compression: gz
dtb:
url: http://.../am335x-bone.dtb
…
- boot:
method: barebox
commands: nfs
prompts:
- 'root@jessie:'
…
The following device-types support barebox:
* ar9331-dpt-module (DPTechnics)
* BeagleBone Black
* dove cubebox (Marvell)
* i.MX23 (Olimex)
* i.MX27-phytec-phycard-s-rdk (PHYTEC)
* i.MX28-duckbill (EnOcean)
* i.MX53-qsbr
* i.MX6dl-riotboard (RIoTboard)
* SoCFPGA Cyclone V SoCrates (Altera)
dfu
---
Allow to use u-boot based dfu instead of the hardware one
On some boards, the hardware dfu serial number is set to 0000000. If
the board provides two boot media (NOR and NAND for example), we can
have a recovery u-boot on one media that will be used to flash the
second media using dfu.
actions:
- deploy:
to: tmpfs
images:
sf_uboot0:
image_arg: --alt sf_uboot0 --download {sf_uboot0}
url: http://.../u-boot-ddr.itb
- boot:
namespace: dfu
method: dfu
Grub
----
When booting with grub, users can now ask to serve the filesystem
from NBD instead of ramdisk or NFS.
- boot:
method: grub
commands: nbd
Minimal
-------
By default LAVA will reset the board power when executing the minimal
boot action. Users can skip this step by adding ``reset: false``. This
can be useful when testing bootloader in interactive tests and then
booting to the OS.
- boot:
method: minimal
reset: false
prompts:
- 'root@debian:~#'
- '/ #'
kexec
-----
The `kexec` boot method has been fixed. Thanks to this support, users
can test this kernel functionality:
- boot:
method: kexec
kernel: /boot/vmlinuz-3.16.0-4-amd64
initrd: /boot/initrd.img-3.16.0-4-amd64
command: /sbin/kexec
kernel-config: /boot/config-3.16.0-4-amd64
boot_message: Starting new kernel
options:
- "--reuse-cmdline"
auto_login:
login_prompt: "login:"
username: root
REST API
========
A new endpoint has been added to list `workers` at
`/api/v0.1/workers`.
Some fields are now using human readable names instead of opaque
constants:
* device:
* health
* state
* device-type:
* health_denominator
* TestJob:
* health
* state
* visibility
* submitter
Job definition schema
=====================
Web UI
------
When submitting or resubmitting a job from the web interface, the new
schema validator will be used. Any errors found by the new schema
validator will be reported as a warning.
Admin helper
------------
A new helper has been created for admins to validate old job
definitions on their local instance:
lava-server manage jobs validate --newer-than 1d
lava-server manage jobs validate --newer-than 1d --submitter lava-health
Job context
-----------
The schema validator is now checking the content of the `context`
dictionary. Only the following keys are now allowed:
* `arch`, `boot_console`, `boot_root`, `cpu`, `extra_options`, `guestfs_driveid`, `guestfs_interface`, `guestfs_size`, `machine`, `memory`, `model`, `monitor`, `netdevice`, `serial`, `vga`
* `bootloader_prompt`, `console_device`, `extra_kernel_args`, `extra_nfsroot_args`, `kernel_loglevel`, `kernel_start_message`, `lava_test_results_dir`, `menu_interrupt_prompt`, `mustang_menu_list`, `test_character_delay`, `tftp_mac_address`
Jobs using keys that are not listed in this list will be rejected.
If you think that some more keys should be whitelisted, please send a
patch.
Docker
======
Debian slim
-----------
The `lava-server` and `lava_dispatcher` docker images are now based on
`debian:stretch-slim`. This will reduce the size of the images
slightly.
lavaserver uid
--------------
In order to prevent any future issues, the `uid` and `gid` of the
`lavaserver` user has been fixed to `200`.
If you are upgrading from a previous lava-server docker image and
keeping the data (job logs, device dictionaries, ...) you will need to
change the owner of the files and directory:
chown -R lavaserver:lavaserver /etc/lava-server/dispatcher-config
chown -R lavaserver:lavaserver /var/lib/lava-server/
Full list of changes
====================
ec8e42c5f schema: vadidate context dictionnary
2e6964c17 device-types: fix board name imx53-qsbr -> imx53-qsrb
52e55a805 Tweak the reprepro-release.sh script again
73ad6ef10 Update the home page contents
d929cc8f9 Upgrade jQuery to v3.4.0 after the last security issue
8dd24244e Revert "device-types: base: detect TFTP ERROR: File not
found"
215a9ca04 Revert "device-types: use array instead of sting
concatenation"
976541d3a Update docs, adding more references to apt.lavasoftware.org
d0e9d7279 GitHelper: when failing, print git command stdout/stderr
e5671dbf1 u-boot-dfu tests: skip if dfu-util is not available
0b660a124 lava shell helpers: fix shellcheck warnings
7f02689c2 Allow to use u-boot based dfu instead of the hardware one
f81e83f31 qemu: Add an ID to the qemu storage
e063102e2 Convert offset and limit query parameters to ints
6363d1020 results: use yaml CDumper instead of the python dumper
d259def87 Fix documentation about template caching
2dbb0e059 device-types: use array instead of sting concatenation
0b3f671bf Fix crash when validating an invalid multinode job
0b6550d6f sample jobs: improve syntax
7c38216ea Schema: remove "live" argument
7e0a8e415 Fix file permissions
562be7cc6 Remove unused allow_modify from grub-nbd sample job
6f25a8086 device-types: add socfpga-cyclone5-socrates device
660b5b126 device-types: add ar9331-dpt-module device
81fa52c07 device-types: add dove-cubox device
65dacd712 device-types: add imx6dl-riotboard device
4566a4639 device-types: add imx53-qsbr device
7b1d2d0b9 device-types: add imx27-phytec-phycard-s-rdk device
2735228e1 device-types: add imx28-duckbill device
91abf4c2c device-types: add imx23-olinuxino device
efec57256 sample_jobs: add barebox job definitions
7b8e78b3c device-types: add parameters section to barebox type
b8102f433 device-types: add helper for barebox test and use on
imx6ul-pico-hobbit
ca800a7b6 xmlrpc: do not raise if jobs.list is empty
705d0aeeb schema: improve inline test definition schema
f5e06bf7d lava-schema: print yaml syntax errors
000420042 schema: allow for auto_login in fastboot and kexec
29708cbb0 schema: allow for no compression
883e24be4 schema: enforce that every actions of a multinode use roles
e9fe783bf Show validation warnings when (re)submitting jobs in the web
ui
c3f8993fe Add NBD support to grub
07a693eaf device-types: minnowboard-common: permit to override
grub_autoboot_prompt
52c9f1ce8 device-types:: allow to change bootloader prompt at device
level
b3864a998 schema: accept zero MultiNode role count
78de67ce0 schema: accept test action parameters
5143b42f3 device-types: add sun5i-a13-olinuxino-micro
7403ddfd4 Remove all links to lavapdu and improve PDUDaemon docs
613038ccd doc: fix typo on --passwd argument
c56ab79e5 schema: add deploy.overlay module
5043f5e83 doc: make the message toward 3rd party packaging more
friendly
0140f42f3 Remove unsed argument
86a923cde Fix version/arch computation in docker debian-slim variants
79d239060 CI: use new sast template provided by gitlab
fce87ba18 cubietruck: use a longer bootloader prompt
83de037f0 Improve sql efficiency when generating lab report
4bab2f601 Improve "lava-server manage check"
714cdb587 Move back adduser call from postinst.py to
lava-server.install
668d132c6 device-types: base: detect TFTP ERROR: File not found
5bf642e3b scheduler: added AM6 device type
eb784fa69 Remove the top-level links from the docs
22f816bbb Add extra doc template code to add rel="canonical" links
95b43d160 boot minimal: allow to skip reseting the device
54f00eea7 Fix kexec boot failure
4819cb9db lava-test-raise: fix shell syntax
d718a4a9d device-type: add meson-g12a-x96-max
0f29a3e9e device-type: add meson-g12a-sei510
386c29ea0 device-type: add new meson-g12-common
affccf663 auto login: fix the prompts list after login
bcb243c65 debian: remove dependencies to unused jquery libs
eece2782b CI: skip debian specific tests if dpkg-query is not
installed
f2fdfa0f9 Improve error message
c974a8859 scheduler: rename rzn1 to rzn1d
b2cebf9a1 Only deploy the LAVA docs on a tag
a7a3f1f2f schema: allow namespace in user commands
f5b6edf73 Use enumerate whenever possible
f44347ce9 Remove "-master" (branch name) in docker tags
e09d68f19 Fix argument typo.
7cd503803 lava_scheduler_app: add barebox test
9ba1c02eb device-types: add imx6ul-pico-hobbit device
9e7a7ecd2 device-types: add base-barebox for typical barebox setup
6fda3991d dispatcher: add barebox boot action
1a6bca180 REST API: translate dictionary values to human readable form
e3b6fde17 Remove /etc/lava-dispatcher/lava-dispatcher.conf
2b0c7569e Upgrade bootstrap to v3.4.1
4cf162244 Use bootstrap minified css file
10396e6a0 release: add the script to push docker containers to docker
hub
2a7e17ab6 check: fix the check for systemd existence
bf4dc683e check: fix a crash when the uid is unknown
443a8b87b Add validate helper to manage jobs support
58eb62334 Remove unused imports
f51d9d000 rest-api: also list workers
2e7cae991 Add --host argument to pg_isready command call
b5090e4d9 debian/lava-dispatcher.dirs: remove usr/sbin
936541ff6 Fix typo in lava-modules.conf install path
Steve, for the LAVA team
--
Steve McIntyre steve.mcintyre(a)linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
Hello,
following the release of LAVA 2019.03, we released lavacli v0.9.5
This release brings one new command: "lavacli jobs validate"
This command send the given job definition to LAVA (version >= 2019.03).
LAVA will check the definition against the schema and report any errors.
As usual, lavacli could be installed from:
* pypi (https://pypi.org/project/lavacli/)
* debian (https://tracker.debian.org/pkg/lavacli) (coming soon).
Cheers
--
Rémi Duraffort
LAVA Team, Linaro
The 2019.03 tag has been pushed to master on git.lavasoftware.org.
Packages have been built in GitLab CI and are available at
apt.lavasoftware.org/release. Docker images for amd64 and arm64 have
been built in GitLab CI and are available from hub.lavasoftware.org
and https://hub.docker.com/u/lavasoftware .
Changes to LAVA UI configuration
================================
https://master.lavasoftware.org/static/docs/v2/advanced-installation.html#c…
Initial settings for a LAVA instance change over time as the
requirements change and dependencies improve internal security
implementations. Most instances will need some adjustment to the
apache configuration for the main LAVA UI in
/etc/apache2/sites-available/lava-server.conf and the LAVA packages do
not attempt to update this file automatically once admins have made
changes. Admins should therefore subscribe to the lava-announce
mailing list and make changes using separate configuration management.
apache2
=======
Browser caching can be improved by enabling mod_header in Apache to
allow LAVA to send appropriate cache control headers as well as mod
proxy and mod proxy_http:
$ sudo a2enmod header
$ sudo a2enmod expires
$ sudo service apache2 restart
LAVA server settings changes
============================
Django requires the allowed hosts to be explicitly set in the LAVA
settings, as a list of hostnames or IP addresses which LAVA is
allowed/expected to use. If this is wrongly configured, the UI will
raise a HTTP 500 error and you will get information in the output of:
lava-server manage check --deploy
or in
/var/log/lava-server/django.log
For example, /etc/lava-server/settings.conf for
https://lava.codehelp.co.uk/contains:
"ALLOWED_HOSTS": ["lava.codehelp.co.uk"],
See also
https://docs.djangoproject.com/en/1.11/ref/settings/#allowed-hosts
It is also important to enable ProxyPreserveHost in
/etc/apache2/sites-available/lava-server.conf
ProxyPreserveHost On
In some situations, you may also need to set USE_X_FORWARDED_HOST to
False in /etc/lava-server/settings.conf
See also
https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-USE_X_FORW…https://git.lavasoftware.org/lava/lava/merge_requests/400
python3-junit.xml
=================
python3-junit.xml >= 1.8 is required for REST API calls when running A
suitable package has now been backported to provide this functionality
for Stretch.
https://packages.debian.org/stretch-backports/python3-junit.xml
Note: there is a very similarly named package which will **not** work
(python3-junitxml). Look for the dot in the package name of the
correct package: python3-junit.xml
When installing, remember to allow stretch-backports:
$ sudo apt -t stretch-backports install python3-junit.xml
https://git.lavasoftware.org/lava/lava/merge_requests/357
Bump version of django-tables2
==============================
With 2019.01 and later, django-tables2 version 1.14 does not pick up
the customisation of the tables, so orderable icons and search forms
and pagination are missing.
It was not practical to backport django-tables2 to stretch-backports
due to complexities in the build dependencies of the package. So this
update has been made available via apt.lavasoftware.org
Stretch users will find a new version of python3-django-tables2 will be
installed alongside 2019.03.
https://git.lavasoftware.org/lava/lava/merge_requests/390
Prevent action.timeout exceeding job.timeout
============================================
This change affects the test jobs you submit to LAVA. The test job will
warn if any one action timeout exceeds the overall job timeout. LAVA is
adopting a stricter submission schema and this is the first common issue.
If you have a repository containing a number of test jobs, you can
check all of the jobs in one go by passing the path to a script in the
lava-common package.
$ sudo apt install lava-common
$ /usr/share/lava-common/lava-schema.py jobs ./*.yaml
You may want to edit the timeouts when resubmitting older test jobs.
When running /usr/share/lava-common/lava-schema.py, there may be a
range of other errors reported too. In future releases, LAVA will no
longer silently ignore unknown or obsolete YAML blocks. If there is
an error in the submission, the expanded schema will report it.
It is worthwhile running /usr/share/lava-common/lava-schema.py
against the output of all your test job submission templates. If you
use templating and CI, then you could add the check to the CI of each
change to the templating. Just ensure that the lava-common package is
upgraded to the same version as the instance to which the test jobs
would be submitted.
In a future release, this expanded schema check will be included into
lavacli, lava-server manage and as an XMLRPC call which can be used
before the submission. For now, only the action.timeout exceeding
job.timeout warning is being enabled in 2019.03.
https://git.lavasoftware.org/lava/lava/merge_requests/413
Changes in this release
=======================
60b9c653f checks: fix crash if the cmdline is not in the expected format
86cfe1ac2 Update reprepro-release.sh
6129fe7c0 tweak reprepro-release for new directory
b0cf8d779 rzn1: allow to boot using a fit image available over tftp
3e4faf6e1 Add qcs404-evb-1k-specific health check and fix a typo
9fd571b3e Fix crash in "check --deploy"
07a89fab3 device-type: Add Qualcomm's qcs404-1k and qcs404-4k
dcf3f2c6c debian: lava meta pkg: allow to use chrony instead of ntp
2d8e09126 (codehelp/master) schema: improve schema validator
8fa55d888 xmlrpc: allow to validate job definition schema
5aa407f04 Add default volumes for lava-slave-docker types
8f2a8cbe6 schema: move validate function to lava_common.schema
0b38b5805 Update copyright years across the LAVA source tree
5c17463e1 Package lava-schema.py in lava-common
a846ad088 Provide symlink for favicon
785906e1b Make timeout check non-fatal
6373b6869 lava-schema: allow to read from stdin
25812ecc7 schema: allow for any objects for test def params
861ec2fb3 Revert change to USE_X_FORWARDED_HOST
f26f61ee7 Avoid spurious boot command list warning
4915231e4 Skip analysis jobs on tags
d53814cc0 Add docs on advanced Apache and container usage
b42520584 Mention lava-lxc-mocker in docs
93828f63d device-types: add Intel upsquare
170dc7547 device-types: add pine64+
f1bbb9bb8 lava-server-gunicorn: allow to set the bind address
354077cb4 Schema: handle recovery deploy and boot methods
ae39f9004 Schema: add deploy to "download" method
7b9119bd7 CI check more sample jobs
168c4db20 Bump version of django-tables2 backport
1b0749092 Fix more sample jobs and ad schroot boot
e30c2f03d Fix typo
ea2987a08 Fix error in maintenance breadcrumb
291aa4ddb Make kernel messages wait sensitive to force_prompt
f3adeb798 Add cls.LT and cls.EXACT to models.DateTimeField
84ad3683a Check health-check YAML and schema
9299d5263 REST API: test /junit
1d5ea3cf7 Require python3-junitxml >= 1.8 for REST API calls
dc5a6fe48 templates: handle multinode vs singlenode
5f01a79c6 CI: run pylint to find common errors
00db03351 Fix some crashes found by pylint3
5a92736ad CI: test templates against the schema
3b5c9d5ef Add extra U-Boot bootloader error
2929ba8f3 Raise the codeclimate complexity threshold
d84756688 apache2: send the right cache-control headers when available
3cc42420f rzn1: do not set uboot_needs_interrupt explicitely
656666cb7 maintenance: fix crash after new state machine
00cea9df4 Device/Worker: remove deprecated method
576146730 Fix missing imports
32960bdd5 Handle duplicate listings in requires
19fdfd969 Move get_domain() to dbutils
01d354d1b img2simg and simg2img now available in a new package
bb62625f3 Fixup changes in the bandit overrides
ae25f1fe1 Update copyright for recent changes
4c33f7a54 Wait for the prompt when using shell on DUT
5662f734a code_quality: install radeo in the setup step
19bda4ac5 Support using radon to output code-climate data
489bc8416 schema: check namespace syntax
6406e1368 Improve inline test schema
0ae489101 Schema: check that jobs with secrets are not public
36d7f4ebb Check that timeouts are shorter than job timeout
fd225060e Improve schema validator
3a2fc459b Prevent action.timeout exceeding job.timeout
e6bfc3f43 Allow check --deploy inside Docker
69053fb82 Ignore DeprecatedWarnings on buster in CI
01fa79469 ci-run: use pytest-3 by default
075bf1f28 Fix crash when pre-test install failed
0c1372f9d Fix crash with older python3-tap versions
02097dd41 Fixup 4ebbc234 - put new test in correct location
feff6c914 CI do not run analyze for tags
9d4a91251 Fix documentation
2354f3899 doc: mention the minimal boot method
354f33ac2 imx8m device type restructuration.
4e1090982 base.jinja2: Allow interrupt-newline to be disabled in u-boot
958068fc6 Fix more sample job syntax
27fdc5107 device-types: add hsdk
dd4663b0a multinode: use character_delay
4ebbc234b power: use run_cmd instead of run_command run_cmd: accept
command as a string
9d9650939 Add missing pytest dependency for autopkgtest
d5bf2c248 Fix typo in f6bb9de2
b3eefdedb Fix missing change which breaks Debian CI testing
> Steve, for the LAVA team
--
Steve McIntyre steve.mcintyre(a)linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
Changes to LAVA UI configuration
================================
https://master.lavasoftware.org/static/docs/v2/advanced-installation.html#c…
Initial settings for a LAVA instance change over time as the
requirements change and dependencies improve internal security
implementations. Most instances will need some adjustment to the
apache configuration for the main LAVA UI in
/etc/apache2/sites-available/lava-server.conf and the LAVA packages do
not attempt to update this file automatically once admins have made
changes. Admins should therefore subscribe to the lava-announce
mailing list and make changes using separate configuration management.
apache2
=======
Browser caching can be improved by enabling mod_header in Apache to
allow LAVA to send appropriate cache control headers as well as mod
proxy and mod proxy_http:
$ sudo a2enmod header
$ sudo a2enmod expires
$ sudo service apache2 restart
LAVA server settings changes
============================
Django requires the allowed hosts to be explicitly set in the LAVA
settings, as a list of hostnames or IP addresses which LAVA is
allowed/expected to use. If this is wrongly configured, the UI will
raise a HTTP 500 error and you will get information in the output of:
lava-server manage check --deploy
or in
/var/log/lava-server/django.log
For example, /etc/lava-server/settings.conf for
https://lava.codehelp.co.uk/ contains:
"ALLOWED_HOSTS": ["lava.codehelp.co.uk"],
See also
https://docs.djangoproject.com/en/1.11/ref/settings/#allowed-hosts
It is also important to enable ProxyPreserveHost in
/etc/apache2/sites-available/lava-server.conf
ProxyPreserveHost On
In some situations, you may also need to set USE_X_FORWARDED_HOST to
False in /etc/lava-server/settings.conf
See also
https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-USE_X_FORW…https://git.lavasoftware.org/lava/lava/merge_requests/400
python3-junit.xml
=================
python3-junit.xml >= 1.8 is required for REST API calls when running A
suitable package has now been backported to provide this functionality
for Stretch.
https://packages.debian.org/stretch-backports/python3-junit.xml
Note: there is a very similarly named package which will **not** work
(python3-junitxml). Look for the full stop in the package name of the
correct package: python3-junit.xml
When installing, remember to allow stretch-backports:
$ sudo apt -t stretch-backports install python3-junit.xml
https://git.lavasoftware.org/lava/lava/merge_requests/357
Bump version of django-tables2
==============================
With 2019.01 and later, django-tables2 version 1.14 does not pick up
the customisation of the tables, so orderable icons and search forms
and pagination are missing.
It was not practical to backport django-tables2 to stretch-backports
due to complexities in the build dependencies of the package. So this
update has been made available via apt.lavasoftware.org
Stretch users will find a new version of python3-django-tables2 will be
installed alongside 2019.02.
https://git.lavasoftware.org/lava/lava/merge_requests/390
Prevent action.timeout exceeding job.timeout
============================================
This change affects the test jobs you submit to LAVA. The test job will
warn if any one action timeout exceeds the overall job timeout. LAVA is
adopting a stricter submission schema and this is the first common issue.
If you have a repository containing a number of test jobs, you can
check all of the jobs in one go by passing the path to a script in the
lava-common package.
$ sudo apt install lava-common
$ /usr/share/lava-common/lava-schema.py jobs ./*.yaml
You may want to edit the timeouts when resubmitting older test jobs.
When running /usr/share/lava-common/lava-schema.py, there may be a
range of other errors reported too. In future releases, LAVA will no
longer silently ignore unknown or obsolete YAML blocks. If there is
an error in the submission, the expanded schema will report it.
It is worthwhile running /usr/share/lava-common/lava-schema.py
against the output of all your test job submission templates. If you
use templating and CI, then you could add the check to the CI of each
change to the templating. Just ensure that the lava-common package is
upgraded to the same version as the instance to which the test jobs
would be submitted.
In the next release, this expanded schema check will be included into
lavacli, lava-server manage and as an XMLRPC call which can be used
before the submission. For now, only the action.timeout exceeding
job.timeout warning is being enabled in 2019.02.
https://git.lavasoftware.org/lava/lava/merge_requests/413
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Hello,
following the release of LAVA 2019.01, we released lavacli v0.9.4
This release brings two new commands:
* "lavacli utils templates render" to render jinja2 templates
This will help users/admins to debug jinja2 templates issues
* "lavacli jobs queue" to show the current queue for the requested
device-types
This is only available since LAVA 2019.01
As usual, lavacli could be installed from:
* pypi (https://pypi.org/project/lavacli/)
* debian (https://tracker.debian.org/pkg/lavacli)
Cheers
--
Rémi Duraffort
LAVA Team, Linaro
The 2019.01 tag has been pushed to master on git.lavasoftware.org.
Packages have been built in GitLab CI and are available at
apt.lavasoftware.org/release. Docker images for amd64 and arm64 have
been built in GitLab CI and are available from hub.lavasoftware.org
and https://hub.docker.com/u/lavasoftware
New bootloader testing and recovery support
==================================
The recovery mode support already implemented for the hikey 6220 has
been used to provide recovery mode operations for the X15 by Milosz
Wasilewski to support bootloader testing with automated recovery.
Dropped default patterns and fixup dicts
===============================
Omits the V1 compatibility constants for default pattern and fixup
dict. Patterns and fix up dictionaries are now only available in Lava
Test Definition 1.0 test shells if the test writer explicitly adds the
parse: block to the test definition. The documentation has been
updated.
Ensure lxc-templates is installed
=========================
To use lxc, the recommended lxc-templates package must be installed to
provide the template files. When installing lava-dispatcher in Docker,
it is now useful to specify the lava-lxc-mocker package as the updated
LXC packages will fail to install inside Docker.
Final black changes
================
All LAVA Software Python code in lava.git has now been formatted using
black and the GitLab CI ensures that this formatting is retained. If
you are developing code to be merged into lava.git, ensure that you
have black installed and run it as the final step before contributing
your changes upstream or the CI will fail.
REST API using django-restframework
=============================
Added REST API for LAVA, including ability to export results in JUnit and TAP13.
Non-POSIX test action
=================
Add a new test method for non-posix shells. The test writter is able
to launch any command on the DUT and is responsible for giving the
right set of patterns that match failure and success.
Add a new boot method called "bootloader"
=================================
This method allows to boot the DUT, interrupt the bootloader and jump
into the bootloader shell.
Changes in this release
==================
ec7875b05 Enhance the Docker admin documentation
ffe2c0751 Build architecture aware docker images
e0c651aae Update docs for development changes
c0eeb4b28 schema: allow for namespaces with interactive tests
aa94f3ac4 Allow to set uboot_needs_interrupt in the device dict
79fb10a12 Fix debian installer deploy method
8b6b53be0 Allow to set fastboot deploy uboot commands
3975eb4f4 doc: add documentation about x15-bl device-type
b87dc6816 scheduler: add x15-bl device type
5d8227c91 scheduler: add recovery support for base-uboot.jinja2
4ddd44ad6 Tweak the development process doc
2bbf5c6e5 Update git repository links
1a9f51494 Drop bashism in vland helper scripts
7ebb3feab Retrieve the vland version in the protocol
f2f933399 Improve docs on health check setup actions
1e596b6ff Improve server|logs|publisher command line parser
68dd8ff79 device-types: allow to set boot_character_delay in device dict
92fa46213 scheduler: randomise the device list
c0c3eb0ef lava-modules.conf should be in /etc/modprobe.d/
2412366c9 Fix sample jobs timeouts
68edafac9 CI: install minicom for mps2plus tests
f3cbdac58 Add lava-server manage command for site
cc16e4b38 mps2: fix validation and documentation
32d18fa3b device-types: add rzn1 template
7a992201c device-types: qemu: Add support for MIPS/PPC/PPC64
e114b354e qemu: Remove root_partition from sample jobs
407d857f9 Fix sample job syntax
049eb5afc sample jobs: move kernel type to the deploy action
25f37dc62 vemsd: recovery_image is mandatory
596aa3b29 Enforce realpath for which check
68b8e54b1 doc: add information about SQUAD to custom-results-handling
1282c4a25 Remove leftover from py2 to py3 conversion
dd76f0797 Add voluptuous to lava-common dependencies
8041e1012 Rework interactive test
f6bb9de2f Improve job page rendering with large jobs
e9fa46977 Improve schema validator and move to lava_common
f116ef3af Add check-devices.py to lava-dev
02e4d9e88 Fix sample jobs syntax
b4d84f82f Drop default patterns and fixup dicts
9fbcca0e4 Separate device and device-type as distinct types
fb4d5785e Action: name is a class variable
e990310e0 Remove deprecated functions since django 1.9
eb25d433e Reorder middlewares according to django default config
ebf17cbe6 Remove deprecated module
8a317b0b2 Record JobError in validate()
3bc657e7b Fix sample jobs syntax
b7eb162a1 Ensure lxc-templates is installed
a3524ce9a Expand minimal configuration for connections
ef2fa8594 Results table: decrease the number of sql queries
4df8e8464 Handle downloads using "Content-Encoding: gzip"
f37c03159 Move sample job to the right directory
37a2fe7fa Remove "connection" from job definition
d70e2623c Add checks for the connection type
5260ae1b9 doc: fix broken links
33918bcb4 Update ./ci-run for lava_common/tests change
5af8543b1 Simplify docker boot command line build
f6be3692c Update emphasis lines of literalinclude blocks
da1d472b5 Fix problems with the purge maintainer script
28afc08eb Remove add-header mentions
832fb942b Use render() shortcut
7ef4ac466 Fix bandit warnings after black formatting
d483d94a5 Make flasher deploy retriable
7f3e23ee1 Do not add yaml_line to the job definition
c76620a55 sample jobs: remove mkimage_arch
5be4be588 Move test_info to the job level
9eb721166 'nfs_url' option does not exist anymore
7c347e04e Fix the increased SAST count from formatting changes
017285ce5 Remove outdated README file
b471ee62f Unify test directory name
46f298251 fix prospector warnings
f7973476b Final black changes for lava_dispatcher
6cae9e765 CI: fix black check
816a47b9b Fix typo in check-devices error message
bc844be5c Apply black to lava_dispatcher
40e6a25f8 Apply black to lava/
93a70fcfc Update restore instructions
4ef3b776a Test repeat fix
01432ba06 Convert test_repeat to pytest
f9e782cc8 Fix repeat logic
72d563661 Update advice on restoring backups
bee7cc6da Update documentation example to use correct syntax
71276203a Fix inline example test job indent error
3d8cd9e88 Advise on upgrading python dependencies
e22dbf0c3 Fix states in maintenance.py
2cf6b51ab interactive: allow to only wait, not sending any command
bfb7cb47c Fix test stages computation when mixing test types
8bdc42ef3 Make StdoutTestCase.pipeline_reference a class method
989b21688 Search the queue by requested_device_type
32a4c821b notification: protect against potential crashes
b7254aec6 XMLRPC-API: decrease the number of sql requests
e65cfdc09 device-types: qemu: Add support for SPARC/SPARC64/MIPS64
ade4b7e0c Apply black to lava_scheduler_app
95aa92f9f Apply black to doc/
54c18a449 Apply black to share/, setup.py and version.py
c74c15d06 Simplify black listing
fd04401a9 Apply black to lava_results_app
8f100d82b Apply black to lava_server
7aa96e077 Apply black to lava_common
5e6e902fa Update documentation for new test action support
bfa253f9b Document the docker deploy
84d202b28 Device table: tag with <i> when worker is undefined.
009e21b53 doc: explain how to serve LAVA under /lava
d86d457f7 device-types: add meson-gxl-s805x-p241
8fb830b3d REST API: allow to export results in JUnit and TAP13
b50800bfc Add a new test method for non-posix shells
ce2627669 Let udev_trigger use the network from udev
172c6f22c Add a new boot method called "bootloader"
4e7248342 Add a qemu-aarch64 device-type
c715f5252 lava_results_app: fix exception when no action_metadata
229242900 Fix deprecation warnings
6cc1c7966 Set ProxyPreserveHost on.
b801a30f4 Create an out-of-tree symlink for djangorestframework
e6678963c Fix rest_framework static symlink
cac2cbb7f Allow dependencies of reprepro-master on schedules
438e23cab Enable reprepro-master
9788877eb checks: log an error when settings.conf is invalid
ad6a2c61d CI: only run deploy jobs for scheduled jobs
f4530df6b TC2 device jinja changes
349ee8175 Remove unused command migrate-lava
6690af7cf CI: skip some tests when lxc-start is not installed
cd7d9e21b Fix crash if dpkg-query does not know about a file
b9cb118ef Add docs on vemsd and mps deployments
d0fd22894 Update the docker-admin page for new links
9bac98720 REST API reduce the number of SQL requests
4bfad2969 Copy lava-lxc-mocker in the publish stage
dacd2a41c Fix lava_rest_app support in ci-run
421e5283d CI: use new sast syntax
0d75252a4 Apply black to lava_server files
b28764d28 Fix unit test exception handling
8e38d322e Add REST API for LAVA
5a55d6646 Update the release repository only on a tag
b55b1c1ba Report the version of tools
3e8f0eb5e Replace with a more pythonic way.
cef780613 Fix doc typo causing build error.
62150f062 Apply black to all migrations
7d040ca09 Optimise to reduce the compare cost.
688df87de Fix: MPS Serial Buffer
8a3d6a587 monitor: fix accepts logic
ade6320a2 Replace the expired job link with a valid one.
bfa558f30 CI build aarch64 docker image using debian 10
ef1c54427 Make navbar and userlinks seperate templates
9270725c9 Correct the path of file "commands.py"
4e7d05696 Tidy up ci-run
73d3a74a4 Ensure sample_jobs validate against the schema
0be0739ab Extend timeouts of example test jobs
b024c2c69 Skip OverlayAction if the overlay is not needed
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Hi,
In founding the LAVA Software Community Project, the team planned to
open up LAVA development more. As already announced by Neil in
September, we have already moved our infrastructure to a GitLab
instance and LAVA developers and users can collaborate there. [1]
The next step in our process is to also open our regular development
design meetings to interested developers. The LAVA design meeting is
where the team gets together to work out deep technical issues, and to
agree on future development goals and ideas. We run these as a weekly
video conference using Google Hangouts Meet [2], We now wish to
welcome other interested developers to join us there too, to help us
develop LAVA.
Summaries of the meetings will be posted regularly to the lava-devel
mailing list [3], and we encourage interested people to subscribe and
discuss LAVA development there.
[1] https://git.lavasoftware.org/
[2] https://meet.google.com/qre-rgen-zwc
[3] https://lists.lavasoftware.org/mailman/listinfo/lava-devel
Cheers,
--
Steve McIntyre steve.mcintyre(a)linaro.org
<http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs
We've done a little bit of advance planning and there are now three
milestones which developers will be using to schedule upcoming
releases.
https://git.lavasoftware.org/groups/lava/-/milestones
Issues and merge requests have been assigned to the relevant milestone.
All dates are approximate.
We will also be announcing (here) when the next release would include
database migrations so that admins have more notice of scheduled
downtime. When a milestone includes database migrations, other changes
which also include a database migration will be moved to that
milestone so that admins can minimise total downtime.
The next release is expected to include the changes from the 2018.12 milestone.
The 2019.02 milestone already includes database migrations.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
