lava-announce

lava-announce@lists.lavasoftware.org

1 participants
134 discussions

2016.8 production release

by Neil Williams

This release moves to requiring django 1.8 from jessie-backports. (django1.7 is no longer supported, django1.8 is the LTS release) See https://lists.linaro.org/pipermail/lava-announce/2016-June/000010.html As part of this release, OpenID and therefore Launchpad login support has been removed - the openid support for django cannot support django1.8. See https://lists.linaro.org/pipermail/lava-announce/2016-June/000011.html This release also includes support for running lava-server with the recently released django1.10 which will soon be available in Debian Stretch (current testing). 2016.8 includes a rewrite or large sections of the documentation and the new logging support for LAVA V2. 2016.8 includes the changes planned for 2016.7, so this is a larger set of changes than previous releases. lava-server changes lava-master: use also get_env_string for multinode Handle unrecognised result messages. Publisher: drop privileges at startup Add lava-publisher init scripts Update mustang jinja template Similar jobs feature. Django1.10 fixes v2: include a 'secrets' field in the job def Adding device-type templates for juno Fix a deprecation warning with render_to_string events: add more details and use a useful username Ensure failed health checks go directly to offline. Faster loading of yaml logs Add a u-boot-commands timeout just for panda Improve error handling in result metadata use job.id inside a not job.is_multinode conditional Fix multinode link from definition back to the job. Allow parentheses in test case names Allow the d02 debian installer grub device to be overridden in device-dictionary Allow for creating devices already offline. Device state transition validation. Open context-sensitive help in a new browser tab Unavailable qemu command should not fail unit tests. LAVA-719 - support branding of source and bugs URL Ensure logging to django logs is info or higher Fix e1d66f to use pk when not multinode. Create and display measurements with units result: don't crash when parsing an invalid result Implement notification blacklist. results: handle skip result Simple notification list. Add 'name' to testcase export. Use the right syntax for character delays First device configuration for ST b2120h410 Fix bug #2278 - inconsistent multinode job id / alias usage Fix HTTP500 by allowing for + in test case names Show job sub_id for multinode jobs. Implement IRC notifications. Fix bug #2263 - parameters and params reference in job def and test def LAVA-708 - Device path should be a list Fix a crash when viewing a query for the first time result: show the metadata as a list (and sublists) Rename conflicting notification properties. log: don't show 'extra' result data log: add a link to each line using AnchorJS log: skip broken strings log: add an icon for the download button log: add link from the result page back to the log Results: improve admin page TestResults: order by job_ids then name Simplify a bit the result page Remove unnecessary loading of django-tables log: redirect complete_log to the job_detail page log: add a link to the result page for each result log: fix HTML syntax errors log: improve rendering of errors and exceptions Update load addresses for larger multiplatform kernels Fix result table log: fix a bug when the page is reloaded Protect from admin error in health check submission Fix default value for device_path to be None and not 'None'. Add missing device_path to nexus jinja templates. log: adapt the result parser to the new log stream mustang UBoot needs 32bit header logs: update job status and device information log: change the arrow when clicking on the affix Fix handling of context with multinode Fix metadata handling for multinode and dynamic connections Fix hidden-device-type listings in JobTable job: remove redudant information Events: add a monitoring thread Initial notifications for v2. lava-master: save the logs in output.yaml job: add a new template for the new log format LAVA-262 Allow admins to expire user accounts log: better formatting of tracebacks Remove support for Django < 1.8 Improve scheduler debug with device details. . Documentation updates Add links and notes to developer branch guide Add notes on making Lava Test Shell portable Add notes on running lava-server unit tests Add timeout documentation. Update the developer guide Document the 'secrets' dictionary Ensure V2 documentation examples are available. update local user account image tidy up api docs Remove multinode use cases tidy up the writing-multinode page expand simple-admin for admin roles tidy up hidden toctree listings for previous/next markup Update chapters for theme Switch to the bootstrap theme updates for multinode and simple administration Major update to the docs for writing multinode tests move all examples into one directory and add test definitions move lava tool issues to a separate file fold the FAQ into the lava-tool docs update the multinode use cases port the mustang example to a separate yaml file use rst macros for see also Add publishing API ref doc initial content for a results intro Move doc yaml to a directory WIP rewrite of the multinode doc Start thinking about how to grow a lab Re-org some early admin stuff Split out the completed YAML jobs Query omit documentation updates. Fix documentation for test definition name handling add instruction for -t jessie-backports move example YAML to an rtsi for easier checking add notes on setting up the first device and device type fix whitespace in migration example Update the scheduling ordering with links Add notes on LAVA being developer focused Update other examples for deploy change fixup deploy action add example of first qemu V2 device start the pipeline design page Minor wording tweaks Rework the hacking session doc expand notes on first installation tweaks and updates for writing tests Fix definition link to log for pipeline Updates for test repositories update multinode docs for V2 fix build messages and errors update examples of params support and custom scripts for parsing complete fixme in advanced-installation add background on CI and LAVA add notes and images for first job submission and results explain the first job and tidy up the example YAML Clean up health check docs add notes for first job Significant cleanup of wording around lava-test-shell Add lots of code-block:: yaml directives Add details of features and architecture. Add content to the what-is section lava-dispatcher changes Only require conversion parameters if using conversion. Support some basic kernel conversion tasks Export the 'secrets' dictionary to the overlay uboot: only log self.errors when it's not empty Enabling NFS only deployment Fix connection_timeout handling for lava-test-shell validate needs to set errors, not raise Fixes for multinode without needing ACK Device environment: fix a crash if env_dut is None Unavailable qemu command should not fail unit tests. Use character_delays instead of character-delays Tests: Use the name defined in the job definition Show when a revision is being applied for VCS Fix bug #2263 - parameters and params reference in job def and test def Improve logging and use lazy logging Log more test results metadata LAVA-708 - Device path should be a list Support kickstart installations on grub, and centos distro type Allow for a prefix with nfsrootfs Finalize: send the job status with the right level log: send the full exception untouched beaglebone-black: update load addresses for larger multiplatform kernels Take into account, device_path that is unset. Fix YAML formatting of wait string log change the result logging format Add sample nexus9 device dictionary. LAVA-701 - Support vendor flashing using fastboot. utils/messages.py: fix out of index error Add support for fixup dictionary and patterns Ensure guest command is available in multinode Stop V2 test shells needing an ACK from the dispatcher logging: add datetime and a better structure Upgrade the failure test case for lavabot First batch of VExpress changes Handle ValueError while waiting To enable jessie-backports, simply copy your existing apt source for jessie and change jessie to jessie-backports, then run apt update. Installations from backports are not automatic, you'll need to tell apt to select jessie-backports for django1.8, as outlined in previous mails to this list: $ sudo apt install -t jessie-backports python-django python-django-common python-django-tables2 Once 2016.8 migrates into Stretch and is then backported to jessie-backports itself, apt will be able to handle the extra dependencies. https://tracker.debian.org/pkg/lava-server https://tracker.debian.org/pkg/lava-dispatcher https://qa.debian.org/developer.php?email=pkg-linaro-lava-devel%40lists.ali… The production repo also has this release. https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava… https://validation.linaro.org/static/docs/v1/installing_on_debian.html#lava… -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

7 years, 8 months

Skipping 2016.7

by Neil Williams

Due to resource shortages in the team, conference time and a few problematic fixes which still need testing, we will have to skip the delayed 2016.7 release and move all changes to the 2016.8 release, due at the end of the first week of August 2016. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

7 years, 9 months

OpenID support to cease with 2016.7

by Neil Williams

OpenID has continued to be available for lava-server on jessie as long as lava-server stayed on django1.7 but the OpenID django support does not operate with django later than 1.7. To continue with upstream django security and bug fix support, lava-server needs to move to requiring django1.8 which means that OpenID can no longer be supported in jessie. Support for OpenID in unstable and testing was removed when django1.8 support first arrived. Once lava-server 2016.7 arrives in jessie-backports, it will conflict with the python-django-auth-openid package - this means that to install the 2016.7 upgrade from jessie-backports, apt will first cause the removal of python-django-auth-openid. (Allowing python-django-auth-openid to remain on the system will corrupt subsequent django operations, causing the package installation to fail during the database migrations.) OpenID support in lava-server is also due to be removed for 2016.7, so the documentation and settings will no longer reference OpenID. LDAP support, Debian SSO and local django accounts will remain available. https://packages.debian.org/stable/python-django-auth-openid https://wiki.debian.org/DebianSingleSignOn -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

7 years, 10 months

Django1.8, Django-Tables and lava-server 2016.7

by Neil Williams

Further to the notice about the availability of django1.8 support: https://lists.linaro.org/pipermail/lava-announce/2016-April/000007.html The 2016.7 release of lava-server will be moving to requiring django1.8 from jessie-backports and will no longer work with django1.7 available in jessie. This has been done because the 1.7 tree is no longer receiving upstream security or bug fix support. lava-server installations newer than 2016.2 can use django1.8 already, as long as python-django-tables2 is updated at the same time. So you are invited to upgrade to django1.8 at any time prior to upgrading to 2016.7. To add jessie-backports, copy the apt source for your favourite Debian mirror and paste that as a new line with jessie updated to jessie-backports. e.g. for the http://mirror.bytemark.co.uk/debian mirror: deb http://mirror.bytemark.co.uk/debian jessie main deb-src http://mirror.bytemark.co.uk/debian jessie main becomes deb http://mirror.bytemark.co.uk/debian jessie main deb http://mirror.bytemark.co.uk/debian jessie-backports main deb-src http://mirror.bytemark.co.uk/debian jessie main $ sudo apt update $ sudo apt install -t jessie-backports python-django python-django-tables2 $ sudo apache2ctl restart $ sudo service lava-server restart It is imperative that python-django-tables2 is updated at the same time as python-django - the old version of django will not work with new tables and the old version of tables will not work with new django. Backports do not automatically replace packages you already have installed, so you remain in control of which backports get installed onto your systems and can run apt dist-upgrade without getting extra upgrades. Backports related to lava-server are also fully supported and are the recommended (soon to be required) way of installing updates for lava-server. After 2016.7, updates of lava-server will not install on jessie without django and django-tables being updated from jessie-backports. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

7 years, 10 months

2016.6 production release

by Neil Williams

lava-server changes * Update V1 docs for Ubuntu changes - lava-server no longer migrates into Ubuntu and was removed from Xenial. * Drop heartbeat support * Prevent scheduler ValueError in reservation * scheduler: reduce the number of SQL queries * Expose DISALLOWED_USER_AGENTS to handle search bots * Add a page for listing Pipeline Devices. * Add Auth support in REST API for more functions * Remove device status glyphicons everywhere, since heartbeat is dead. * Create metadata on the number of test definitions * Remove the need for extensions * Remove deprecated lava_projects * Update docs for guestfs and resulting issues. * Enable job definition metadata. * dispatcher-master: support zmq CURVE encryption * Add documentation on using ZMQ curve lava-dispatcher changes * Load the overlay as an extra QEMU drive to prevent need for loopback devices. * Support delays between sending characters for all actions * LXC support for fastboot and adb and drop the need for adb or fastboot to be installed. * Remove adb related code * Port V2 unit tests only to python3 * ZMQ: Support encryption of the logs sent to the server * Add support for QEMU Debian Installer tests * Update standards version (no changes) * Remove adb and fastboot from recommends and add python-guestfs as a dependency. To use LXC with LAVA 2016.6 (V2 only), the lxc package itself needs to be installed from jessie-backports (the lava-dispatcher package has a version requirement in the Recommends which enforces this, if lxc is installed). Additionally, for the new version of LXC to work correctly, the dispatcher itself needs to be rebooted to apply the LXC updates to the kernel boot arguments. To enable jessie-backports, simply copy your existing apt source for jessie and change jessie to jessie-backports, then run apt update. Installations from backports are not automatic, you'll need to tell apt to select jessie-backports for the LXC package: apt install -t jessie-backports lxc https://validation.linaro.org/ has already been updated to 2016.6. Packages have been uploaded to Debian (unstable) and will appear on the mirrors in due course. Once these versions migrate into stretch, backports will be made to jessie-backports. https://tracker.debian.org/pkg/lava-server https://tracker.debian.org/pkg/lava-dispatcher https://qa.debian.org/developer.php?email=pkg-linaro-lava-devel%40lists.ali… The production repo also has this release. https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava… https://validation.linaro.org/static/docs/v1/installing_on_debian.html#lava… -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

7 years, 10 months

New GuestFS dependency added in 2016.6 for V2

by Neil Williams

The 2016.6 release which is currently in preparation adds some important dependencies which some admins may prefer to install ahead of the upgrade. Debian has the concept of Dependencies which must be installed and Recommends which are optional but expected to be useful by most users of the package in question. The Recommends for GuestFS can be omitted from the installation if admins desire but this needs to be done ahead of the upgrade to 2016.6. V2 in 2016.6 adds support for GuestFS which removes the need for loopback support for QEMU devices. GuestFS supports a wide range of filesystem utilities and other virtualisation support tools, not all of which will be of interest to LAVA. (Notably, libguestfs Recommends a mail-transport-agent and the default MTA in Debian is exim. With the default exim4 config, this simply adds a local MTA with no external support.) Debian supports opting out of Recommends when installing packages, so if admins have concerns about extra packages being installed on the dispatchers (e.g. if using ARMv7 dispatchers or simply to reduce the complexity of the install) then Recommends can be omitted for the installation of these dependencies: The option is: $ sudo apt --no-install-recommends install python-guestfs We'll add a note to the installation docs for this too. With the move to V2 having a dumb dispatcher, the dependencies themselves do need to be installed to allow the server to dictate what jobs the dispatcher can run without the dispatcher failing due to missing support. If later LAVA upgrades need any of the packages that would have been installed as Recommends, LAVA will add an explicit dependency on just those packages and any others will remain optional. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

7 years, 10 months

2016.4 backports uploaded - a note on django and django-tables

by Neil Williams

Updates of existing backports are not usually announced in this way but there is a note to consider, so I'm covering that here. Independent of the update of the lava-server and lava-dispatcher backports to 2016.4, admins using jessie-backports do need to be aware of a issue which is still pending related to django and django-tables. A backport of django-tables is pending, waiting for these two packages to also be accepted into jessie-backports: https://ftp-master.debian.org/new/django-haystack_2.4.1-1~bpo8%2B1.html https://ftp-master.debian.org/new/python-fudge_1.1.0-1~bpo8%2B1.html These are required before django-tables can build inside jessie-backports and therefore before I can upload a build of django-tables for jessie-backports. Irrespective of the version of LAVA being used, if django itself is installed from jessie-backports, the version of django-tables in jessie will cause a HTTP500. The fix is to either downgrade django back to the version in jessie or to pull in the version of python-django-tables2 from *testing*.: https://packages.debian.org/stretch/python-django-tables2. It is this upstream version which is to be backported to jessie-backports. Once these last two dependencies clear the queue for backports, I'll make the backport of django-tables and then admins can have django and django-tables from jessie-backports which will work. i.e. the installed django needs to match up with the installed django-tables - either both from jessie or, once available, both from jessie-backports. This isn't a problem in LAVA as such, it affects LAVA and I'll have the fix available just as soon as the queue clears within Debian. Finally, LAVA is likely to move to using django from jessie-backports once this issue is fixed so that instances gain the security fixes in the django 1.8 release - at that point, LAVA will ensure that the updated django-tables is pulled in at the same time. LAVA has been running with django 1.8 (and django1.9) for developers for some time, testing with 1.8 will continue in the meantime. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

7 years, 12 months

2016.4 production release

by Neil Williams

lava-server changes: Add support for python-django-debug-toolbar Deleting V1 filters now cascade delete image chart filters. Reduce the number of SQL queries used on common pages. Improve scheduler iterative performance. Add support for deleting unused tokens Stop runaway healthchecks in V2. Migrate option_list to argparse for django 1.8 and later. Allow authentication with result export in V2 Drop references to Ubuntu beyond 2016.9.post1 Implement omitting individual results from queries in V2 Indicate omitted results and allow including them back in. Add a management command for refreshing queries Change V1 measurement field to be float only. Clean up top-level documentation Introduce limit to queries in V2. * Suggest python-django-debug-toolbar * Refresh all V2 queries during package postinst to ensure materialized views are available. lava-dispatcher changes: Allow for adb not being available on some arches Support ramdisks that are not compressed Move the ramdisk header requirements into the device template Fix calling the Bzr unit tests Use tar flags from deployment data when unpacking overlay kvm-aarch32: introduce device type for 32-bit guest on 64-bit KVM host kvm-aarch64: expose a virtual random number generator (RNG) to the guest Provide an apache2 config for V2 workers Initial support for device using GRUB bootloader in V2 Add in the ability to halt the V1 master image before powering off No longer require ssh options or identity_file Don't use -p port in SCP options https://validation.linaro.org/ will be updated in due course (probably at the start of next week). Packages have been uploaded to Debian (unstable) and will appear on the mirrors in due course. Once these versions migrate into stretch, backports can be made to jessie-backports. https://tracker.debian.org/pkg/lava-server https://tracker.debian.org/pkg/lava-dispatcher https://qa.debian.org/developer.php?email=pkg-linaro-lava-devel%40lists.ali… The production repo also has this release. https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava… https://validation.linaro.org/static/docs/v1/installing_on_debian.html#lava… Information will also appear at http://releases.linaro.org/components/lava/ in due course. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years

Deprecating the old dispatcher

by Neil Williams

The 2016.2 release marks the start of the migration to the pipeline dispatcher. The previous dispatcher code, the documentation for that code and the support for JSON job formats are now *deprecated*. As a result, the corresponding lava-server code has also been *deprecated*, this includes Bundles, Filters and Image Reports (original and 2.0). The deprecated Dashboard is replaced by Results with Queries and Charts which have been ported to the data produced by the pipeline testjobs. Future development within LAVA will be based solely on the pipeline dispatcher and associated server side objects. Support for the deprecated dispatcher will be maintained for future releases only during the migration and *will be removed* once the migration is complete. The migration is expected to take most of 2016. Individual devices and particular items of job support will be gradually disabled until the migration completes for all devices and all jobs. The migration process involves: 0: Adding pipeline support for devices and deployment methods 1: Migrating user submissions and automated submissions to the pipeline support. For the Linaro LAVA instances, this will happen within the particular teams inside Linaro. 2: Making selected devices exclusive to the pipeline so that JSON jobs can no longer be submitted for those devices. At this point, support for JSON jobs which rely on those devices will cease. 3: Once all devices are exclusive, reject all JSON submissions. 4: Disable the old scheduler daemon on the LAVA instance. At some point, probably in 2017 - a month or two after JSON submissions cease - the migration will complete by executing: 5: Removal of code support for the old dispatcher in lava-dispatcher and lava-tool codebases and associated documentation. 6: Removal of code support for database objects specific to the old dispatcher, like Bundle, in lava-server with associated documentation. This will involve the deletion of the Bundle data as well as image reports, filters and BundleStreams. TestJob objects which are not pipeline jobs will also be deleted. 7: Removal of the rest of the code which is still dependent upon or only used to support deprecated objects and functions. 8: Modification of code which only exists to isolate the deprecated objects from the pipeline objects. e.g. newly created jobs or devices will default to pipeline support. LAVA instances outside Linaro will need to manage their own migrations during 2016 if updates are to be applied during 2017. It is not possible to retain the database objects without the deprecated code, so owners of individual instances may choose to create an archive instance which has no online devices, accepts no submissions, just provides access to a snapshot of the data at the time that JSON submissions ceased. To maintain access to the archived data, this instance must not upgrade to LAVA releases after the archive is created and the original devices should persist in the database but be kept in the Offline state. Devices should also have submissions restricted to a single administrator account for which no API token exists. More information on the details of the migration and the state of support for jobs using the old dispatcher will be announced on this mailing list. As a reminder - lava-announce is a read-only list. Posts to this list are only made by the LAVA software team. Replies need to be directed to linaro-validation(a)lists.linaro.org - your email client should do this for you, using the Reply-To header added by mailman. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 2 months

Issues with Django OpenID Authentication

by Neil Williams

Please reply to linaro-validation(a)lists.linaro.org Please let us know if you are using OpenID authentication with LAVA. Newer versions of django will make it impossible to support django-openid-auth in Debian unstable and testing. The version of django-openid-auth in Jessie can continue to be used, so we would like to know how many users want to continue with this support. OpenID as a protocol has been dying for some time and Linaro has moved over to LDAP, which is fine if LDAP is already available. The time pressure for this change is coming from the schedule to get the latest django and the latest lava packages into Ubuntu Xenial 16.04LTS which means that support needs to be implemented in the 2015.12 or 2016.1 LAVA releases. This is why this is quickly following the trusty change. We have been aware of the issues with django-openid-auth for some time, it was only when we had completed the move of the Cambridge lab to LDAP that changes involving django-openid-auth could be considered. If you are using OpenID authentication (e.g. using Launchpad or Google OpenID), please let us know. If you would like to see some other forms of authentication supported, also let us know. We can investigate Python Social Auth (http://psa.matiasaguirre.net/), if there is interest. If we don't hear from users who want django-openid-auth support for use on Debian Jessie, we will drop django-openid-auth support from all lava builds. This will leave LDAP and local Django accounts in 2015.12. If anyone has experience of other django authentication modules, also let us know. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 4 months

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

lava-announce