This release moves to requiring django 1.8 from jessie-backports.
(django1.7 is no longer supported, django1.8 is the LTS release)
See https://lists.linaro.org/pipermail/lava-announce/2016-June/000010.html
As part of this release, OpenID and therefore Launchpad login support
has been removed - the openid support for django cannot support
django1.8.
See https://lists.linaro.org/pipermail/lava-announce/2016-June/000011.html
This release also includes support for running lava-server with the
recently released django1.10 which will soon be available in Debian
Stretch (current testing).
2016.8 includes a rewrite or large sections of the documentation and
the new logging support for LAVA V2.
2016.8 includes the changes planned for 2016.7, so this is a larger
set of changes than previous releases.
lava-server changes
lava-master: use also get_env_string for multinode
Handle unrecognised result messages.
Publisher: drop privileges at startup
Add lava-publisher init scripts
Update mustang jinja template
Similar jobs feature.
Django1.10 fixes
v2: include a 'secrets' field in the job def
Adding device-type templates for juno
Fix a deprecation warning with render_to_string
events: add more details and use a useful username
Ensure failed health checks go directly to offline.
Faster loading of yaml logs
Add a u-boot-commands timeout just for panda
Improve error handling in result metadata
use job.id inside a not job.is_multinode conditional
Fix multinode link from definition back to the job.
Allow parentheses in test case names
Allow the d02 debian installer grub device to be overridden
in device-dictionary
Allow for creating devices already offline.
Device state transition validation.
Open context-sensitive help in a new browser tab
Unavailable qemu command should not fail unit tests.
LAVA-719 - support branding of source and bugs URL
Ensure logging to django logs is info or higher
Fix e1d66f to use pk when not multinode.
Create and display measurements with units
result: don't crash when parsing an invalid result
Implement notification blacklist.
results: handle skip result
Simple notification list.
Add 'name' to testcase export.
Use the right syntax for character delays
First device configuration for ST b2120h410
Fix bug #2278 - inconsistent multinode job id / alias usage
Fix HTTP500 by allowing for + in test case names
Show job sub_id for multinode jobs.
Implement IRC notifications.
Fix bug #2263 - parameters and params reference in job def and test def
LAVA-708 - Device path should be a list
Fix a crash when viewing a query for the first time
result: show the metadata as a list (and sublists)
Rename conflicting notification properties.
log: don't show 'extra' result data
log: add a link to each line using AnchorJS
log: skip broken strings
log: add an icon for the download button
log: add link from the result page back to the log
Results: improve admin page
TestResults: order by job_ids then name
Simplify a bit the result page
Remove unnecessary loading of django-tables
log: redirect complete_log to the job_detail page
log: add a link to the result page for each result
log: fix HTML syntax errors
log: improve rendering of errors and exceptions
Update load addresses for larger multiplatform kernels
Fix result table
log: fix a bug when the page is reloaded
Protect from admin error in health check submission
Fix default value for device_path to be None and not 'None'.
Add missing device_path to nexus jinja templates.
log: adapt the result parser to the new log stream
mustang UBoot needs 32bit header
logs: update job status and device information
log: change the arrow when clicking on the affix
Fix handling of context with multinode
Fix metadata handling for multinode and dynamic connections
Fix hidden-device-type listings in JobTable
job: remove redudant information
Events: add a monitoring thread
Initial notifications for v2.
lava-master: save the logs in output.yaml
job: add a new template for the new log format
LAVA-262 Allow admins to expire user accounts
log: better formatting of tracebacks
Remove support for Django < 1.8
Improve scheduler debug with device details.
.
Documentation updates
Add links and notes to developer branch guide
Add notes on making Lava Test Shell portable
Add notes on running lava-server unit tests
Add timeout documentation.
Update the developer guide
Document the 'secrets' dictionary
Ensure V2 documentation examples are available.
update local user account image
tidy up api docs
Remove multinode use cases
tidy up the writing-multinode page
expand simple-admin for admin roles
tidy up hidden toctree listings for previous/next markup
Update chapters for theme
Switch to the bootstrap theme
updates for multinode and simple administration
Major update to the docs for writing multinode tests
move all examples into one directory and add test definitions
move lava tool issues to a separate file
fold the FAQ into the lava-tool docs
update the multinode use cases
port the mustang example to a separate yaml file
use rst macros for see also
Add publishing API ref doc
initial content for a results intro
Move doc yaml to a directory
WIP rewrite of the multinode doc
Start thinking about how to grow a lab
Re-org some early admin stuff
Split out the completed YAML jobs
Query omit documentation updates.
Fix documentation for test definition name handling
add instruction for -t jessie-backports
move example YAML to an rtsi for easier checking
add notes on setting up the first device and device type
fix whitespace in migration example
Update the scheduling ordering with links
Add notes on LAVA being developer focused
Update other examples for deploy change
fixup deploy action
add example of first qemu V2 device
start the pipeline design page
Minor wording tweaks
Rework the hacking session doc
expand notes on first installation
tweaks and updates for writing tests
Fix definition link to log for pipeline
Updates for test repositories
update multinode docs for V2
fix build messages and errors
update examples of params support and custom scripts for parsing
complete fixme in advanced-installation
add background on CI and LAVA
add notes and images for first job submission and results
explain the first job and tidy up the example YAML
Clean up health check docs
add notes for first job
Significant cleanup of wording around lava-test-shell
Add lots of code-block:: yaml directives
Add details of features and architecture.
Add content to the what-is section
lava-dispatcher changes
Only require conversion parameters if using conversion.
Support some basic kernel conversion tasks
Export the 'secrets' dictionary to the overlay
uboot: only log self.errors when it's not empty
Enabling NFS only deployment
Fix connection_timeout handling for lava-test-shell
validate needs to set errors, not raise
Fixes for multinode without needing ACK
Device environment: fix a crash if env_dut is None
Unavailable qemu command should not fail unit tests.
Use character_delays instead of character-delays
Tests: Use the name defined in the job definition
Show when a revision is being applied for VCS
Fix bug #2263 - parameters and params reference in job def and test def
Improve logging and use lazy logging
Log more test results metadata
LAVA-708 - Device path should be a list
Support kickstart installations on grub, and centos distro type
Allow for a prefix with nfsrootfs
Finalize: send the job status with the right level
log: send the full exception untouched
beaglebone-black: update load addresses for larger multiplatform kernels
Take into account, device_path that is unset.
Fix YAML formatting of wait string log
change the result logging format
Add sample nexus9 device dictionary.
LAVA-701 - Support vendor flashing using fastboot.
utils/messages.py: fix out of index error
Add support for fixup dictionary and patterns
Ensure guest command is available in multinode
Stop V2 test shells needing an ACK from the dispatcher
logging: add datetime and a better structure
Upgrade the failure test case for lavabot
First batch of VExpress changes
Handle ValueError while waiting
To enable jessie-backports, simply copy your existing apt source for
jessie and change jessie to jessie-backports, then run apt update.
Installations from backports are not automatic, you'll need to tell
apt to select jessie-backports for django1.8, as outlined in previous
mails to this list:
$ sudo apt install -t jessie-backports python-django
python-django-common python-django-tables2
Once 2016.8 migrates into Stretch and is then backported to
jessie-backports itself, apt will be able to handle the extra
dependencies.
https://tracker.debian.org/pkg/lava-serverhttps://tracker.debian.org/pkg/lava-dispatcherhttps://qa.debian.org/developer.php?email=pkg-linaro-lava-devel%40lists.ali…
The production repo also has this release.
https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…https://validation.linaro.org/static/docs/v1/installing_on_debian.html#lava…
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Due to resource shortages in the team, conference time and a few
problematic fixes which still need testing, we will have to skip the
delayed 2016.7 release and move all changes to the 2016.8 release, due
at the end of the first week of August 2016.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
OpenID has continued to be available for lava-server on jessie as long
as lava-server stayed on django1.7 but the OpenID django support does
not operate with django later than 1.7. To continue with upstream
django security and bug fix support, lava-server needs to move to
requiring django1.8 which means that OpenID can no longer be supported
in jessie. Support for OpenID in unstable and testing was removed when
django1.8 support first arrived.
Once lava-server 2016.7 arrives in jessie-backports, it will conflict
with the python-django-auth-openid package - this means that to
install the 2016.7 upgrade from jessie-backports, apt will first cause
the removal of python-django-auth-openid. (Allowing
python-django-auth-openid to remain on the system will corrupt
subsequent django operations, causing the package installation to fail
during the database migrations.)
OpenID support in lava-server is also due to be removed for 2016.7, so
the documentation and settings will no longer reference OpenID.
LDAP support, Debian SSO and local django accounts will remain available.
https://packages.debian.org/stable/python-django-auth-openidhttps://wiki.debian.org/DebianSingleSignOn
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Further to the notice about the availability of django1.8 support:
https://lists.linaro.org/pipermail/lava-announce/2016-April/000007.html
The 2016.7 release of lava-server will be moving to requiring
django1.8 from jessie-backports and will no longer work with django1.7
available in jessie.
This has been done because the 1.7 tree is no longer receiving
upstream security or bug fix support.
lava-server installations newer than 2016.2 can use django1.8 already,
as long as python-django-tables2 is updated at the same time. So you
are invited to upgrade to django1.8 at any time prior to upgrading to
2016.7.
To add jessie-backports, copy the apt source for your favourite Debian
mirror and paste that as a new line with jessie updated to
jessie-backports.
e.g. for the http://mirror.bytemark.co.uk/debian mirror:
deb http://mirror.bytemark.co.uk/debian jessie main
deb-src http://mirror.bytemark.co.uk/debian jessie main
becomes
deb http://mirror.bytemark.co.uk/debian jessie main
deb http://mirror.bytemark.co.uk/debian jessie-backports main
deb-src http://mirror.bytemark.co.uk/debian jessie main
$ sudo apt update
$ sudo apt install -t jessie-backports python-django python-django-tables2
$ sudo apache2ctl restart
$ sudo service lava-server restart
It is imperative that python-django-tables2 is updated at the same
time as python-django - the old version of django will not work with
new tables and the old version of tables will not work with new
django.
Backports do not automatically replace packages you already have
installed, so you remain in control of which backports get installed
onto your systems and can run apt dist-upgrade without getting extra
upgrades.
Backports related to lava-server are also fully supported and are the
recommended (soon to be required) way of installing updates for
lava-server.
After 2016.7, updates of lava-server will not install on jessie
without django and django-tables being updated from jessie-backports.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
lava-server changes
* Update V1 docs for Ubuntu changes - lava-server no longer
migrates into Ubuntu and was removed from Xenial.
* Drop heartbeat support
* Prevent scheduler ValueError in reservation
* scheduler: reduce the number of SQL queries
* Expose DISALLOWED_USER_AGENTS to handle search bots
* Add a page for listing Pipeline Devices.
* Add Auth support in REST API for more functions
* Remove device status glyphicons everywhere, since heartbeat is dead.
* Create metadata on the number of test definitions
* Remove the need for extensions
* Remove deprecated lava_projects
* Update docs for guestfs and resulting issues.
* Enable job definition metadata.
* dispatcher-master: support zmq CURVE encryption
* Add documentation on using ZMQ curve
lava-dispatcher changes
* Load the overlay as an extra QEMU drive to prevent
need for loopback devices.
* Support delays between sending characters for all actions
* LXC support for fastboot and adb and drop the need for adb or
fastboot to be installed.
* Remove adb related code
* Port V2 unit tests only to python3
* ZMQ: Support encryption of the logs sent to the server
* Add support for QEMU Debian Installer tests
* Update standards version (no changes)
* Remove adb and fastboot from recommends and add python-guestfs as a
dependency.
To use LXC with LAVA 2016.6 (V2 only), the lxc package itself needs to
be installed from jessie-backports (the lava-dispatcher package has a
version requirement in the Recommends which enforces this, if lxc is
installed). Additionally, for the new version of LXC to work
correctly, the dispatcher itself needs to be rebooted to apply the LXC
updates to the kernel boot arguments.
To enable jessie-backports, simply copy your existing apt source for
jessie and change jessie to jessie-backports, then run apt update.
Installations from backports are not automatic, you'll need to tell
apt to select jessie-backports for the LXC package:
apt install -t jessie-backports lxc
https://validation.linaro.org/ has already been updated to 2016.6.
Packages have been uploaded to Debian (unstable) and will appear on
the mirrors in due course. Once these versions migrate into stretch,
backports will be made to jessie-backports.
https://tracker.debian.org/pkg/lava-serverhttps://tracker.debian.org/pkg/lava-dispatcherhttps://qa.debian.org/developer.php?email=pkg-linaro-lava-devel%40lists.ali…
The production repo also has this release.
https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…https://validation.linaro.org/static/docs/v1/installing_on_debian.html#lava…
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
The 2016.6 release which is currently in preparation adds some
important dependencies which some admins may prefer to install ahead
of the upgrade.
Debian has the concept of Dependencies which must be installed and
Recommends which are optional but expected to be useful by most users
of the package in question. The Recommends for GuestFS can be omitted
from the installation if admins desire but this needs to be done ahead
of the upgrade to 2016.6.
V2 in 2016.6 adds support for GuestFS which removes the need for
loopback support for QEMU devices. GuestFS supports a wide range of
filesystem utilities and other virtualisation support tools, not all
of which will be of interest to LAVA. (Notably, libguestfs Recommends
a mail-transport-agent and the default MTA in Debian is exim. With the
default exim4 config, this simply adds a local MTA with no external
support.)
Debian supports opting out of Recommends when installing packages, so
if admins have concerns about extra packages being installed on the
dispatchers (e.g. if using ARMv7 dispatchers or simply to reduce the
complexity of the install) then Recommends can be omitted for the
installation of these dependencies:
The option is:
$ sudo apt --no-install-recommends install python-guestfs
We'll add a note to the installation docs for this too.
With the move to V2 having a dumb dispatcher, the dependencies
themselves do need to be installed to allow the server to dictate what
jobs the dispatcher can run without the dispatcher failing due to
missing support. If later LAVA upgrades need any of the packages that
would have been installed as Recommends, LAVA will add an explicit
dependency on just those packages and any others will remain optional.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Updates of existing backports are not usually announced in this way
but there is a note to consider, so I'm covering that here.
Independent of the update of the lava-server and lava-dispatcher
backports to 2016.4, admins using jessie-backports do need to be aware
of a issue which is still pending related to django and django-tables.
A backport of django-tables is pending, waiting for these two packages
to also be accepted into jessie-backports:
https://ftp-master.debian.org/new/django-haystack_2.4.1-1~bpo8%2B1.htmlhttps://ftp-master.debian.org/new/python-fudge_1.1.0-1~bpo8%2B1.html
These are required before django-tables can build inside
jessie-backports and therefore before I can upload a build of
django-tables for jessie-backports.
Irrespective of the version of LAVA being used, if django itself is
installed from jessie-backports, the version of django-tables in
jessie will cause a HTTP500. The fix is to either downgrade django
back to the version in jessie or to pull in the version of
python-django-tables2 from *testing*.:
https://packages.debian.org/stretch/python-django-tables2. It is this
upstream version which is to be backported to jessie-backports.
Once these last two dependencies clear the queue for backports, I'll
make the backport of django-tables and then admins can have django and
django-tables from jessie-backports which will work. i.e. the
installed django needs to match up with the installed django-tables -
either both from jessie or, once available, both from
jessie-backports.
This isn't a problem in LAVA as such, it affects LAVA and I'll have
the fix available just as soon as the queue clears within Debian.
Finally, LAVA is likely to move to using django from jessie-backports
once this issue is fixed so that instances gain the security fixes in
the django 1.8 release - at that point, LAVA will ensure that the
updated django-tables is pulled in at the same time. LAVA has been
running with django 1.8 (and django1.9) for developers for some time,
testing with 1.8 will continue in the meantime.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
lava-server changes:
Add support for python-django-debug-toolbar
Deleting V1 filters now cascade delete image chart filters.
Reduce the number of SQL queries used on common pages.
Improve scheduler iterative performance.
Add support for deleting unused tokens
Stop runaway healthchecks in V2.
Migrate option_list to argparse for django 1.8 and later.
Allow authentication with result export in V2
Drop references to Ubuntu beyond 2016.9.post1
Implement omitting individual results from queries in V2
Indicate omitted results and allow including them back in.
Add a management command for refreshing queries
Change V1 measurement field to be float only.
Clean up top-level documentation
Introduce limit to queries in V2.
* Suggest python-django-debug-toolbar
* Refresh all V2 queries during package postinst to ensure
materialized views are available.
lava-dispatcher changes:
Allow for adb not being available on some arches
Support ramdisks that are not compressed
Move the ramdisk header requirements into the device template
Fix calling the Bzr unit tests
Use tar flags from deployment data when unpacking overlay
kvm-aarch32: introduce device type for 32-bit guest on 64-bit KVM
host
kvm-aarch64: expose a virtual random number generator (RNG) to the
guest
Provide an apache2 config for V2 workers
Initial support for device using GRUB bootloader in V2
Add in the ability to halt the V1 master image before powering off
No longer require ssh options or identity_file
Don't use -p port in SCP options
https://validation.linaro.org/ will be updated in due course (probably
at the start of next week).
Packages have been uploaded to Debian (unstable) and will appear on
the mirrors in due course. Once these versions migrate into stretch,
backports can be made to jessie-backports.
https://tracker.debian.org/pkg/lava-serverhttps://tracker.debian.org/pkg/lava-dispatcherhttps://qa.debian.org/developer.php?email=pkg-linaro-lava-devel%40lists.ali…
The production repo also has this release.
https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…https://validation.linaro.org/static/docs/v1/installing_on_debian.html#lava…
Information will also appear at
http://releases.linaro.org/components/lava/ in due course.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
The 2016.2 release marks the start of the migration to the pipeline
dispatcher. The previous dispatcher code, the documentation for that
code and the support for JSON job formats are now *deprecated*. As a
result, the corresponding lava-server code has also been *deprecated*,
this includes Bundles, Filters and Image Reports (original and 2.0).
The deprecated Dashboard is replaced by Results with Queries and
Charts which have been ported to the data produced by the pipeline
testjobs.
Future development within LAVA will be based solely on the pipeline
dispatcher and associated server side objects. Support for the
deprecated dispatcher will be maintained for future releases only
during the migration and *will be removed* once the migration is
complete. The migration is expected to take most of 2016. Individual
devices and particular items of job support will be gradually disabled
until the migration completes for all devices and all jobs.
The migration process involves:
0: Adding pipeline support for devices and deployment methods
1: Migrating user submissions and automated submissions to the
pipeline support. For the Linaro LAVA instances, this will happen
within the particular teams inside Linaro.
2: Making selected devices exclusive to the pipeline so that JSON jobs
can no longer be submitted for those devices. At this point, support
for JSON jobs which rely on those devices will cease.
3: Once all devices are exclusive, reject all JSON submissions.
4: Disable the old scheduler daemon on the LAVA instance.
At some point, probably in 2017 - a month or two after JSON
submissions cease - the migration will complete by executing:
5: Removal of code support for the old dispatcher in lava-dispatcher
and lava-tool codebases and associated documentation.
6: Removal of code support for database objects specific to the old
dispatcher, like Bundle, in lava-server with associated documentation.
This will involve the deletion of the Bundle data as well as image
reports, filters and BundleStreams. TestJob objects which are not
pipeline jobs will also be deleted.
7: Removal of the rest of the code which is still dependent upon or
only used to support deprecated objects and functions.
8: Modification of code which only exists to isolate the deprecated
objects from the pipeline objects. e.g. newly created jobs or devices
will default to pipeline support.
LAVA instances outside Linaro will need to manage their own migrations
during 2016 if updates are to be applied during 2017.
It is not possible to retain the database objects without the
deprecated code, so owners of individual instances may choose to
create an archive instance which has no online devices, accepts no
submissions, just provides access to a snapshot of the data at the
time that JSON submissions ceased. To maintain access to the archived
data, this instance must not upgrade to LAVA releases after the
archive is created and the original devices should persist in the
database but be kept in the Offline state. Devices should also have
submissions restricted to a single administrator account for which no
API token exists.
More information on the details of the migration and the state of
support for jobs using the old dispatcher will be announced on this
mailing list.
As a reminder - lava-announce is a read-only list. Posts to this list
are only made by the LAVA software team. Replies need to be directed
to
linaro-validation(a)lists.linaro.org - your email client should do this
for you, using the Reply-To header added by mailman.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Please reply to linaro-validation(a)lists.linaro.org
Please let us know if you are using OpenID authentication with LAVA.
Newer versions of django will make it impossible to support
django-openid-auth in Debian unstable and testing. The version of
django-openid-auth in Jessie can continue to be used, so we would like
to know how many users want to continue with this support.
OpenID as a protocol has been dying for some time and Linaro has moved
over to LDAP, which is fine if LDAP is already available.
The time pressure for this change is coming from the schedule to get
the latest django and the latest lava packages into Ubuntu Xenial
16.04LTS which means that support needs to be implemented in the
2015.12 or 2016.1 LAVA releases. This is why this is quickly following
the trusty change. We have been aware of the issues with
django-openid-auth for some time, it was only when we had completed
the move of the Cambridge lab to LDAP that changes involving
django-openid-auth could be considered.
If you are using OpenID authentication (e.g. using Launchpad or Google
OpenID), please let us know.
If you would like to see some other forms of authentication supported,
also let us know. We can investigate Python Social Auth
(http://psa.matiasaguirre.net/), if there is interest.
If we don't hear from users who want django-openid-auth support for
use on Debian Jessie, we will drop django-openid-auth support from all
lava builds. This will leave LDAP and local Django accounts in
2015.12.
If anyone has experience of other django authentication modules, also
let us know.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
