lava-announce

lava-announce@lists.lavasoftware.org

147 discussions

2016.9 production release

by Neil Williams

2016.9 includes a rewrite of each of the daemons used by LAVA, lava-server, lava-master and lava-slave. When installing the upgrade, admins need to accept the maintainer's version of the scripts in /etc/init.d/ for these daemons. Where previously changes were made in the scripts in /etc/init.d/, the same parameters can now be changed in associated configuration files to prevent these conflicts in the future. /etc/init.d/lava-slave - change /etc/lava-dispatcher/lava-slave /etc/init.d/lava-master - change /etc/lava-server/lava-master /etc/init.d/lava-server - change /etc/default/lava-server The prompt would look something like this: Configuration file '/etc/init.d/lava-server' ==> Modified (by you or by a script) since installation. ==> Package distributor has shipped an updated version. What would you like to do about it ? Your options are: Y or I : install the package maintainer's version N or O : keep your currently-installed version D : show the differences between the versions Z : start a shell to examine the situation The default action is to keep your current version. *** lava-server (Y/I/N/O/D/Z) [default=N] ? y See https://validation.linaro.org/static/docs/v2/pipeline-server.html#configuri… References in the changelogs, commit history and announcements which start with LAVA- followed by a number relate to the planning stories which the LAVA team use for new features. The LAVA planning has now been opened up to anonymous access by appending the reference to a stub URL: http://projects.linaro.org/browse/, e.g. http://projects.linaro.org/browse/LAVA-62 The planning stories do not support comments or changes other than by the LAVA software team. If you have comments on the stories, please use the lava-users mailing list. lava-server changes Show measurements and units of testcase detail page Drop lxc schema - no support for alternatives Configure logging for linaro_django_xmlpc calls Fix link to lava-tool context-help Fix the "Debian-based distribution" link grammar Improve monitor test type, allow multiple tests LAVA-494 allow searching for device tags in tables Add support for LXC with multinode Fix broken ref link for unit_test Fix NoReverseMatch from ImageReports2.0 editing. Fix typo in ./share/validate.py --hostname Update hikey jinja2 templates for LXC Fix occasionally failing multinode tests. Add a configuration file for lava-master Update lava-slave and lava-master docs LAVA-374 - drop versiontools Tidy up some build errors and reformat consistently. Improve documentation describing inline test definitions Major improvements for the "Writing MultiNode" page Add docs for the lava-vland-names change Using BOOTX in Juno bootcmds Correcting default uboot commands in juno device template Document the code locations for developers Ensure multinode description is generated correctly Allowing overrides to connection and action timeouts master: Fix log rotations Remove filtering by filter id for available testcases in image reports. Init: remove the right lock file and simplify Adding uboot jinja2 templates for all Juno flavours LAVA-739 - declare sub_id and job.id for multinode master: don't validate jobs on the master select_device only accepts pipeline jobs Remove legacy SyntaxHighlighter from job definition pages. lava-master: simplify log file handling Fix title disappearance on image reports. Ignore retired devices in validate call Don't add anchors to logs for running jobs Don't crash when log contains invalid test names Schema: accept 'parse' for inline test definitions Use libYAML when loading logs LAVA queries use cases. Bug link fix for url regex. Add LogEntry for BugLinks. Update the init scripts Remove the entry point as it's no longer needed Ensure test-case name is valid Extend vland unit tests to cover assignment Expand on the device dictionary and templates Expand the notes on using jessie-backports Add a Contents page Rewrite lava-server command line from scratch events: set umask to a restrictive value Bug links port. LAVA-200 - track admin actions made in the UI Add device tags support to V2 Allow the monitor test type Update qemu command line options for arm64 lava-dispatcher changes Catch the edge case where serial line corruption means we miss the testrun Add support for alternative Test actions Update manpage of lava-slave for configuration change. Allow configuring the slave from file git: use -C instead of --git-dir Allow setting the envrionment on most systems Handle exceptions from subprocess from compression download: don't expand the path when retrying Grab finalize from the root pipeline Log exception as strings and not objects LAVA-740 - export vlan names and interface names Reopen log file when rotating the logs LAVA-699 - Support hi6220-hikey board with lxc Init: remove the right lock file and simplify pipeline:vland: raise JobError if switch_id or port_id is None Remove redundant assignation Allow logging at the validate stage Force the shell to produce an initial prompt Handle errors if the testrun start is omitted. Pipeline parser: sort and reverse in one call logs: remove unused support for local logs slave: put all tmp files into the same directory Power: improve logging message Exclude patterns from description Port lava-slave to Python3 ApplyOverlayTftp: remove duplicated untar_file ExportDeviceEnvironment: only source valid shell_file's add LEDE to available distributions Always log the command to be run beforehand. There is currently a delay on uploading lava-server to jessie-backports due to problems with uwsgi in Debian. A backport will be made available once these problems are fixed. In the meantime, the same build as will later appear in jessie-backports is available in the production-repo: https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava… https://tracker.debian.org/pkg/lava-server https://tracker.debian.org/pkg/lava-dispatcher https://qa.debian.org/developer.php?email=pkg-linaro-lava-devel%40lists.ali… -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

10 years

Configuration file updates in 2016.9

by Neil Williams

A number of the configuration files which manage the daemons used by lava-server and the lava-slave daemon for lava-dispatcher are being updated in 2016.9. This has implications for those who install the upgrade, especially if this is normally done by automation. Configuration file '/etc/init.d/lava-slave' ==> Modified (by you or by a script) since installation. ==> Package distributor has shipped an updated version. What would you like to do about it ? Your options are: Y or I : install the package maintainer's version N or O : keep your currently-installed version D : show the differences between the versions Z : start a shell to examine the situation The default action is to keep your current version. *** lava-slave (Y/I/N/O/D/Z) [default=N] ? y Installing new version of config file /etc/init.d/lava-slave ... dpkg will check the checksums of the init files against the current and previous packages and halt installation with the above prompt in case of doubt. Admins may need to install the update manually (directly on the command line). The recommendation is to accept the package maintainer's version but all admins may want to check the difference between the two. Changes in this release include fixing the log rotate handling of lava-master using reload and a rewrite of the lava-server CLI script to drop the use of entrypoints which changes the syntax of the daemon options. The most common reason for init files to be modified is to change the loglevel. This can be done in /etc/default/lava-server: # Additional options that are passed to the Daemon. DAEMON_OPTS="loglevel=debug" -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

10 years

Community access to LAVA planning

by Neil Williams

LAVA has long been developed in the open as far as the git repositories and bug reports are concerned. However, the git commit history can often include references to LAVA-NNN and these references have previously been obscure. Anonymous access has now been granted to all LAVA-* references, so here's some context on what and where. https://projects.linaro.org/projects/LAVA This is a JIRA instance and we refer to issues in JIRA as cards or stories. All stories in LAVA are now visible to the community, so you can map any of the LAVA-NNN references directly to the actual URL by prepending projects.linaro.org/browse/LAVA-NNN, e.g. https://projects.linaro.org/browse/LAVA-734 It has not been possible to open the actual organisation of the stories (KANBAN) and some elements do not provide the same links as when viewed internally but hopefully some of those issues can be addressed in the future. For now, community members can follow any explicit LAVA-NNN reference from the git history or elsewhere to the actual story and view the complete description and all comments on that story. Most stories also provide a direct link to the review which implemented that story within the Comments section. Merged reviews provide information in the "Included In" menu about which branches and tags include that commit. Tags match the production release of the same name as the tag, so this allows everyone to know which releases contain which changes. Stories which are currently status Open may simply be ideas which still need to be scoped. Some of these will be dropped without being implemented as use cases and code development moves on. Dates in stories are very rough estimates, usually longer than it will actually require. If community members have comments or questions on particular stories, please use the lava-users mailing list. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

10 years

2016.8 production release

by Neil Williams

This release moves to requiring django 1.8 from jessie-backports. (django1.7 is no longer supported, django1.8 is the LTS release) See https://lists.linaro.org/pipermail/lava-announce/2016-June/000010.html As part of this release, OpenID and therefore Launchpad login support has been removed - the openid support for django cannot support django1.8. See https://lists.linaro.org/pipermail/lava-announce/2016-June/000011.html This release also includes support for running lava-server with the recently released django1.10 which will soon be available in Debian Stretch (current testing). 2016.8 includes a rewrite or large sections of the documentation and the new logging support for LAVA V2. 2016.8 includes the changes planned for 2016.7, so this is a larger set of changes than previous releases. lava-server changes lava-master: use also get_env_string for multinode Handle unrecognised result messages. Publisher: drop privileges at startup Add lava-publisher init scripts Update mustang jinja template Similar jobs feature. Django1.10 fixes v2: include a 'secrets' field in the job def Adding device-type templates for juno Fix a deprecation warning with render_to_string events: add more details and use a useful username Ensure failed health checks go directly to offline. Faster loading of yaml logs Add a u-boot-commands timeout just for panda Improve error handling in result metadata use job.id inside a not job.is_multinode conditional Fix multinode link from definition back to the job. Allow parentheses in test case names Allow the d02 debian installer grub device to be overridden in device-dictionary Allow for creating devices already offline. Device state transition validation. Open context-sensitive help in a new browser tab Unavailable qemu command should not fail unit tests. LAVA-719 - support branding of source and bugs URL Ensure logging to django logs is info or higher Fix e1d66f to use pk when not multinode. Create and display measurements with units result: don't crash when parsing an invalid result Implement notification blacklist. results: handle skip result Simple notification list. Add 'name' to testcase export. Use the right syntax for character delays First device configuration for ST b2120h410 Fix bug #2278 - inconsistent multinode job id / alias usage Fix HTTP500 by allowing for + in test case names Show job sub_id for multinode jobs. Implement IRC notifications. Fix bug #2263 - parameters and params reference in job def and test def LAVA-708 - Device path should be a list Fix a crash when viewing a query for the first time result: show the metadata as a list (and sublists) Rename conflicting notification properties. log: don't show 'extra' result data log: add a link to each line using AnchorJS log: skip broken strings log: add an icon for the download button log: add link from the result page back to the log Results: improve admin page TestResults: order by job_ids then name Simplify a bit the result page Remove unnecessary loading of django-tables log: redirect complete_log to the job_detail page log: add a link to the result page for each result log: fix HTML syntax errors log: improve rendering of errors and exceptions Update load addresses for larger multiplatform kernels Fix result table log: fix a bug when the page is reloaded Protect from admin error in health check submission Fix default value for device_path to be None and not 'None'. Add missing device_path to nexus jinja templates. log: adapt the result parser to the new log stream mustang UBoot needs 32bit header logs: update job status and device information log: change the arrow when clicking on the affix Fix handling of context with multinode Fix metadata handling for multinode and dynamic connections Fix hidden-device-type listings in JobTable job: remove redudant information Events: add a monitoring thread Initial notifications for v2. lava-master: save the logs in output.yaml job: add a new template for the new log format LAVA-262 Allow admins to expire user accounts log: better formatting of tracebacks Remove support for Django < 1.8 Improve scheduler debug with device details. . Documentation updates Add links and notes to developer branch guide Add notes on making Lava Test Shell portable Add notes on running lava-server unit tests Add timeout documentation. Update the developer guide Document the 'secrets' dictionary Ensure V2 documentation examples are available. update local user account image tidy up api docs Remove multinode use cases tidy up the writing-multinode page expand simple-admin for admin roles tidy up hidden toctree listings for previous/next markup Update chapters for theme Switch to the bootstrap theme updates for multinode and simple administration Major update to the docs for writing multinode tests move all examples into one directory and add test definitions move lava tool issues to a separate file fold the FAQ into the lava-tool docs update the multinode use cases port the mustang example to a separate yaml file use rst macros for see also Add publishing API ref doc initial content for a results intro Move doc yaml to a directory WIP rewrite of the multinode doc Start thinking about how to grow a lab Re-org some early admin stuff Split out the completed YAML jobs Query omit documentation updates. Fix documentation for test definition name handling add instruction for -t jessie-backports move example YAML to an rtsi for easier checking add notes on setting up the first device and device type fix whitespace in migration example Update the scheduling ordering with links Add notes on LAVA being developer focused Update other examples for deploy change fixup deploy action add example of first qemu V2 device start the pipeline design page Minor wording tweaks Rework the hacking session doc expand notes on first installation tweaks and updates for writing tests Fix definition link to log for pipeline Updates for test repositories update multinode docs for V2 fix build messages and errors update examples of params support and custom scripts for parsing complete fixme in advanced-installation add background on CI and LAVA add notes and images for first job submission and results explain the first job and tidy up the example YAML Clean up health check docs add notes for first job Significant cleanup of wording around lava-test-shell Add lots of code-block:: yaml directives Add details of features and architecture. Add content to the what-is section lava-dispatcher changes Only require conversion parameters if using conversion. Support some basic kernel conversion tasks Export the 'secrets' dictionary to the overlay uboot: only log self.errors when it's not empty Enabling NFS only deployment Fix connection_timeout handling for lava-test-shell validate needs to set errors, not raise Fixes for multinode without needing ACK Device environment: fix a crash if env_dut is None Unavailable qemu command should not fail unit tests. Use character_delays instead of character-delays Tests: Use the name defined in the job definition Show when a revision is being applied for VCS Fix bug #2263 - parameters and params reference in job def and test def Improve logging and use lazy logging Log more test results metadata LAVA-708 - Device path should be a list Support kickstart installations on grub, and centos distro type Allow for a prefix with nfsrootfs Finalize: send the job status with the right level log: send the full exception untouched beaglebone-black: update load addresses for larger multiplatform kernels Take into account, device_path that is unset. Fix YAML formatting of wait string log change the result logging format Add sample nexus9 device dictionary. LAVA-701 - Support vendor flashing using fastboot. utils/messages.py: fix out of index error Add support for fixup dictionary and patterns Ensure guest command is available in multinode Stop V2 test shells needing an ACK from the dispatcher logging: add datetime and a better structure Upgrade the failure test case for lavabot First batch of VExpress changes Handle ValueError while waiting To enable jessie-backports, simply copy your existing apt source for jessie and change jessie to jessie-backports, then run apt update. Installations from backports are not automatic, you'll need to tell apt to select jessie-backports for django1.8, as outlined in previous mails to this list: $ sudo apt install -t jessie-backports python-django python-django-common python-django-tables2 Once 2016.8 migrates into Stretch and is then backported to jessie-backports itself, apt will be able to handle the extra dependencies. https://tracker.debian.org/pkg/lava-server https://tracker.debian.org/pkg/lava-dispatcher https://qa.debian.org/developer.php?email=pkg-linaro-lava-devel%40lists.ali… The production repo also has this release. https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava… https://validation.linaro.org/static/docs/v1/installing_on_debian.html#lava… -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

10 years, 1 month

Skipping 2016.7

by Neil Williams

Due to resource shortages in the team, conference time and a few problematic fixes which still need testing, we will have to skip the delayed 2016.7 release and move all changes to the 2016.8 release, due at the end of the first week of August 2016. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

10 years, 1 month

OpenID support to cease with 2016.7

by Neil Williams

OpenID has continued to be available for lava-server on jessie as long as lava-server stayed on django1.7 but the OpenID django support does not operate with django later than 1.7. To continue with upstream django security and bug fix support, lava-server needs to move to requiring django1.8 which means that OpenID can no longer be supported in jessie. Support for OpenID in unstable and testing was removed when django1.8 support first arrived. Once lava-server 2016.7 arrives in jessie-backports, it will conflict with the python-django-auth-openid package - this means that to install the 2016.7 upgrade from jessie-backports, apt will first cause the removal of python-django-auth-openid. (Allowing python-django-auth-openid to remain on the system will corrupt subsequent django operations, causing the package installation to fail during the database migrations.) OpenID support in lava-server is also due to be removed for 2016.7, so the documentation and settings will no longer reference OpenID. LDAP support, Debian SSO and local django accounts will remain available. https://packages.debian.org/stable/python-django-auth-openid https://wiki.debian.org/DebianSingleSignOn -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

10 years, 2 months

Django1.8, Django-Tables and lava-server 2016.7

by Neil Williams

Further to the notice about the availability of django1.8 support: https://lists.linaro.org/pipermail/lava-announce/2016-April/000007.html The 2016.7 release of lava-server will be moving to requiring django1.8 from jessie-backports and will no longer work with django1.7 available in jessie. This has been done because the 1.7 tree is no longer receiving upstream security or bug fix support. lava-server installations newer than 2016.2 can use django1.8 already, as long as python-django-tables2 is updated at the same time. So you are invited to upgrade to django1.8 at any time prior to upgrading to 2016.7. To add jessie-backports, copy the apt source for your favourite Debian mirror and paste that as a new line with jessie updated to jessie-backports. e.g. for the http://mirror.bytemark.co.uk/debian mirror: deb http://mirror.bytemark.co.uk/debian jessie main deb-src http://mirror.bytemark.co.uk/debian jessie main becomes deb http://mirror.bytemark.co.uk/debian jessie main deb http://mirror.bytemark.co.uk/debian jessie-backports main deb-src http://mirror.bytemark.co.uk/debian jessie main $ sudo apt update $ sudo apt install -t jessie-backports python-django python-django-tables2 $ sudo apache2ctl restart $ sudo service lava-server restart It is imperative that python-django-tables2 is updated at the same time as python-django - the old version of django will not work with new tables and the old version of tables will not work with new django. Backports do not automatically replace packages you already have installed, so you remain in control of which backports get installed onto your systems and can run apt dist-upgrade without getting extra upgrades. Backports related to lava-server are also fully supported and are the recommended (soon to be required) way of installing updates for lava-server. After 2016.7, updates of lava-server will not install on jessie without django and django-tables being updated from jessie-backports. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

10 years, 2 months

2016.6 production release

by Neil Williams

lava-server changes * Update V1 docs for Ubuntu changes - lava-server no longer migrates into Ubuntu and was removed from Xenial. * Drop heartbeat support * Prevent scheduler ValueError in reservation * scheduler: reduce the number of SQL queries * Expose DISALLOWED_USER_AGENTS to handle search bots * Add a page for listing Pipeline Devices. * Add Auth support in REST API for more functions * Remove device status glyphicons everywhere, since heartbeat is dead. * Create metadata on the number of test definitions * Remove the need for extensions * Remove deprecated lava_projects * Update docs for guestfs and resulting issues. * Enable job definition metadata. * dispatcher-master: support zmq CURVE encryption * Add documentation on using ZMQ curve lava-dispatcher changes * Load the overlay as an extra QEMU drive to prevent need for loopback devices. * Support delays between sending characters for all actions * LXC support for fastboot and adb and drop the need for adb or fastboot to be installed. * Remove adb related code * Port V2 unit tests only to python3 * ZMQ: Support encryption of the logs sent to the server * Add support for QEMU Debian Installer tests * Update standards version (no changes) * Remove adb and fastboot from recommends and add python-guestfs as a dependency. To use LXC with LAVA 2016.6 (V2 only), the lxc package itself needs to be installed from jessie-backports (the lava-dispatcher package has a version requirement in the Recommends which enforces this, if lxc is installed). Additionally, for the new version of LXC to work correctly, the dispatcher itself needs to be rebooted to apply the LXC updates to the kernel boot arguments. To enable jessie-backports, simply copy your existing apt source for jessie and change jessie to jessie-backports, then run apt update. Installations from backports are not automatic, you'll need to tell apt to select jessie-backports for the LXC package: apt install -t jessie-backports lxc https://validation.linaro.org/ has already been updated to 2016.6. Packages have been uploaded to Debian (unstable) and will appear on the mirrors in due course. Once these versions migrate into stretch, backports will be made to jessie-backports. https://tracker.debian.org/pkg/lava-server https://tracker.debian.org/pkg/lava-dispatcher https://qa.debian.org/developer.php?email=pkg-linaro-lava-devel%40lists.ali… The production repo also has this release. https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava… https://validation.linaro.org/static/docs/v1/installing_on_debian.html#lava… -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

10 years, 3 months

New GuestFS dependency added in 2016.6 for V2

by Neil Williams

The 2016.6 release which is currently in preparation adds some important dependencies which some admins may prefer to install ahead of the upgrade. Debian has the concept of Dependencies which must be installed and Recommends which are optional but expected to be useful by most users of the package in question. The Recommends for GuestFS can be omitted from the installation if admins desire but this needs to be done ahead of the upgrade to 2016.6. V2 in 2016.6 adds support for GuestFS which removes the need for loopback support for QEMU devices. GuestFS supports a wide range of filesystem utilities and other virtualisation support tools, not all of which will be of interest to LAVA. (Notably, libguestfs Recommends a mail-transport-agent and the default MTA in Debian is exim. With the default exim4 config, this simply adds a local MTA with no external support.) Debian supports opting out of Recommends when installing packages, so if admins have concerns about extra packages being installed on the dispatchers (e.g. if using ARMv7 dispatchers or simply to reduce the complexity of the install) then Recommends can be omitted for the installation of these dependencies: The option is: $ sudo apt --no-install-recommends install python-guestfs We'll add a note to the installation docs for this too. With the move to V2 having a dumb dispatcher, the dependencies themselves do need to be installed to allow the server to dictate what jobs the dispatcher can run without the dispatcher failing due to missing support. If later LAVA upgrades need any of the packages that would have been installed as Recommends, LAVA will add an explicit dependency on just those packages and any others will remain optional. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

10 years, 3 months

2016.4 backports uploaded - a note on django and django-tables

by Neil Williams

Updates of existing backports are not usually announced in this way but there is a note to consider, so I'm covering that here. Independent of the update of the lava-server and lava-dispatcher backports to 2016.4, admins using jessie-backports do need to be aware of a issue which is still pending related to django and django-tables. A backport of django-tables is pending, waiting for these two packages to also be accepted into jessie-backports: https://ftp-master.debian.org/new/django-haystack_2.4.1-1~bpo8%2B1.html https://ftp-master.debian.org/new/python-fudge_1.1.0-1~bpo8%2B1.html These are required before django-tables can build inside jessie-backports and therefore before I can upload a build of django-tables for jessie-backports. Irrespective of the version of LAVA being used, if django itself is installed from jessie-backports, the version of django-tables in jessie will cause a HTTP500. The fix is to either downgrade django back to the version in jessie or to pull in the version of python-django-tables2 from *testing*.: https://packages.debian.org/stretch/python-django-tables2. It is this upstream version which is to be backported to jessie-backports. Once these last two dependencies clear the queue for backports, I'll make the backport of django-tables and then admins can have django and django-tables from jessie-backports which will work. i.e. the installed django needs to match up with the installed django-tables - either both from jessie or, once available, both from jessie-backports. This isn't a problem in LAVA as such, it affects LAVA and I'll have the fix available just as soon as the queue clears within Debian. Finally, LAVA is likely to move to using django from jessie-backports once this issue is fixed so that instances gain the security fixes in the django 1.8 release - at that point, LAVA will ensure that the updated django-tables is pulled in at the same time. LAVA has been running with django 1.8 (and django1.9) for developers for some time, testing with 1.8 will continue in the meantime. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

10 years, 4 months

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

lava-announce