Hi folks,

The 2019.05 tag has been pushed to master on git.lavasoftware.org. .deb packages have been built in GitLab CI and are available at

https://apt.lavasoftware.org/release

Docker images for amd64 and arm64 have been built in GitLab CI and will be available shortly from

https://hub.lavasoftware.org/

and

https://hub.docker.com/u/lavasoftware

Changes in this release =======================

Device-types ============

Aliases -------

The device-type aliases feature has been totally redesigned. You can now define multiple aliases for a given device-type. In a job definition, users can use either the device-type or any of its corresponding aliases.

New device-types ----------------

New supported devices:

* da850-lcdk (davinci) * SoCA9 (Schneider)

frdm-k64f ---------

It is now possible to override the setting `resets_after_flash` in the device dictionary, to support newer firmwares.

meson-gl12 ----------

Fix the dtb and ramdisk addresses: * dtb: **0x01070000** (was **0x0107f000**) * ramdisk: **0x08000000** (was **0x04000000**)

RPi ---

Move the ramdisk load address to **0x02200000** (was **0x02100000**) to allow more space for the kernel image and avoid them overlapping.

rzn1d -----

Allow to use zImage.

Database ========

Migrations ----------

This new version creates several database migrations to: * remove `TestJob.admin_notifications` field * improve `DeviceType` aliases * replace some CharField types with TextField to remove unnecessary size limitations

These may incur a small amount of downtime during the upgrade process as database migrations happen. A goal for the LAVA development team is to reduce the frequency of database migrations to reduce this cost as much as possible. The next **planned** set of database migrations is in the 2019.08 release, in August.

dashboard_app -------------

The dashboard_app (holding v1 results) has been fully removed, including: * migrations * database model * xmlrpc calls

Docker ======

multiarch ---------

Due to some limitations in the docker registry embedded in GitLab, we are no longer building and providing multiarch docker images for every commit. Multiarch images will still be available on [docker hub](https://hub.docker.com/u/lavasoftware/) for every release.

lava-dispatcher ---------------

The dispatcher docker image now includes `docker-ce-cli`. This will allow lava to control docker containers from within a lava-dispatcher container, helping admins to participate in lavafed.

Server performances ===================

Template caching ----------------

The `USE_TEMPLATE_CACHE` setting variable has been removed. On newer Django versions, this option is no longer useful and might even cause problems.

Compressing job logs --------------------

Support has been added for compressing log files on the server, using xz (lzma) compression. Logs will be stored uncompressed as jobs are running, but they may later be compressed by admins to reduce the amount of space needed.

The `lava-server manage jobs compress` command includes options to allow for management of this. Admins can use this in cronjobs to automatically compress job logs older than 30 days, or similar (`lava-server manage jobs compress --older-than 30d --slow`).

LAVA will automatically decompress old compressed logs on the fly if they are ever needed again in future.

Job definition schema =====================

Admin helper ------------

The `lava-server manage jobs validate` command can now send a mail to the admin when an invalid job is found.

To make use of this, admins can setup a regular cronjob that will call: ```shell lava-server manage jobs validate --mail-admins ```

Job context -----------

The schema validator is now checking the content of the `context` dictionary for both single and multinode jobs.

Two new allowed keys have been added: `uboot_extra_error_message` and `uboot_needs_interrupt`

Only the following keys are now allowed: * `arch`, `boot_console`, `boot_root`, `cpu`, `extra_options`, `guestfs_driveid`, `guestfs_interface`, `guestfs_size`, `machine`, `memory`, `model`, `monitor`, `netdevice`, `serial`, `vga` * `bootloader_prompt`, `console_device`, `extra_kernel_args`, `extra_nfsroot_args`, `kernel_loglevel`, `kernel_start_message`, `lava_test_results_dir`, `menu_interrupt_prompt`, `mustang_menu_list`, `test_character_delay`, `tftp_mac_address`, `uboot_extra_error_message`, `uboot_needs_interrupt`

Jobs using keys that are not listed in this list will be rejected.

If you think that some more keys should be whitelisted, talk to us.

Extending job context ---------------------

Admins can extend the white list by updating `EXTRA_CONTEXT_VARIABLES` in the settings:

```json "EXTRA_CONTEXT_VARIABLES": ["custom_variable1", "variable_2"] ```

Don't forget to restart `lava-server-gunicorn` if you make changes here.

lava-slave ==========

Socks proxy -----------

When connecting with `lava-master` and `lava-logs`, `lava-slave` can now use a socks proxy if needed.

In order to use a socks proxy, you should either: * start lava-slave with `--socks-proxy localhost:8081` * edit `/etc/lava-dispatcher/lava-slave` and add `SOCKS_PROXY="--socks-proxy localhost:8081"`

Custom CSS ==========

This adds the possibility to include a custom style sheet in a LAVA instance for applying overall branding to the GUI.

Add the link to the CSS in the `settings.conf` and restart `lava-server-gunicorn`: ```json "BRANDING_CSS": "https://www.example.org/css/style.css", ```

Full list of changes since 2019.04 ================================== 5e4a36956 jobs compress: create the output.yaml.size file if needed. 412b73788 device-type: add SoCA9 from Schneider aab54ce66 Revert "Auth refactoring part 1." 396c7342f logutils: test the module functions 15bb54214 logutils: only create the index when needed ed60e521b Add a command line helper to compress logs e19c9a2cf Read logs from either output.yaml or output.yaml.xz 41672feee job logs: replace TestJob.output_file by logutils functions c3bf98bd6 job: remove unused/uneeded job_file_present variable c5c881347 Remove the archived_job_file function 1e32ed252 LAVA branding: add possibility to include custom CSS 417c8b14b release: add a script to remove old docker images 0e08fa08e Clean up obsolete conffiles causing piuparts failures 1e9778114 Present the codeclimate data in human readable format 2fa918740 schema: the job visibility is required. 2fbc72031 Order aliases by name when rendering 1a10f4903 scheduler.jobs.validate: fix return value e7ce9d63c schema: allow admins to extend the white list of context variable a615696cb lava-schema: factorize and allow to check device config 0c68b065c schema: add a device schema into lava_common 1ac122efe Add fastboot support to the meson-g12a-sei510 56583c703 debian: remove unused or redondant files 9654c526a Remove permission object support from GroupObjectPermission methods. 8d13f2def Change API to update single object permissions. f365005b1 Small code review changes. 444404943 Add LAVAServerError exception tree. 7c533d5c4 Remove shortened codename support. c991c3c5c Auth refactoring part 1. c2acdbdd8 debian: remove unused py3dist overrides dadce92a6 doc: fix sample job schema c505f1322 schema: add missing uboot_needs_interrupt to the context ccfbf2aed schema: fix boot.minimal by adding reset and auto_login b17b24e94 schema: also validate the multinode context variable 0f13bc60a Change notification CharField types to TextField instead 4be183e2d unittests.sh was renamed to dispatcher-unittests.sh. d6b642471 Partially revert 3d0ddb402 to fix some tables issues ff613da21 device: improve the recent test jobs table e21f2ddbb Use .exists() instead of len(.all()) bb3ca4525 XMLRPC API: fix crash in scheduler.aliases 85fc254b7 schema: fix remaining syntax error in the sample jobs 9cccb2e3a docs: Update docs for DeviceType alias 5b6424218 scheduler: add test for alias submissions 62c0e08ec scheduler/api: fix XML-RPC API for Aliases ec74e9ed7 scheduler: avoid device type-alias name conflict 3d0ddb402 scheduler: modify alias to be 1-many relation a6e9c6d2f lava-slave: add an option to use a socks proxy 7a18fc022 schema: allow uboot_extra_error_message in the context 711a20131 schema: fix definition of test definition option "lava-signal" b0088485c doc: fix emphasis aefc87b26 Improve the README and use markdown for Gitlab rendering a4b350516 fix pyocd boot action method to print out version of binary used c045401d1 Do not use minified version of JS and CSS files 55afae484 schema: add "new_connection" boot method ed4eeda0a doc: add missing label 851b4bc33 version.py: fix git check when using worktrees 72b9dd18c Change the 'fastboot_sequence' 383525981 user commands: use run_cmd instead of run_command b1c84a1d6 validate: allow to report invalid jobs to admins 82aa705b8 Fix multinode jobs validation b10bc826f Use the static template function d48ba02c6 Make USB/DFU/UDEV timeouts into Infra Errors 643180264 debian: list every minimun versions for buster 0dd5dc1d6 lava_common: remove unused dependency on pyzmq f1c1372ca doc: fix syntax to match schema 759740e1c device-types: rpi: allow extra space for kernel image 3bd8f0d8a Fix file permissions 8139950b1 Remove USE_TEMPLATE_CACHE option 44c5e6ca5 scheduler: update rzn1d device-type to boot zImage 3fced42f5 Fix the dtb/ramdisk addresses 750d8f298 Document the "command" action block 0447e8327 doc: replace lava-tool by lavacli whenever possible 8c73a28ed Update packaging for dashboard removal 1c6be7f71 Remove leftover from v1 dashboard cc7e86fa7 Remove dashboard_app c4a4c3e3b device-type: add da850-lcdk 3a16ca954 Remove admin_notifications field 1a20620be CI: do not build anymore multiarch images 796c98eb3 CI: fix tag selection cc2bc7509 add systemd support for lava-os-build a7c483bdb Add overriding resets_after_flash to support newer FRDM-K64F boards 537773c9e TestJob duration: return only seconds (and not miliseconds) 5ae7f5a1a Add a helper script for Debian release 4b2c07da7 Fix broken seealso syntax in docs 447bc9344 doc/v2/timeouts: fix http-download action name typos

Steve, for the LAVA team