The 2016.2 release marks the start of the migration to the pipeline
dispatcher. The previous dispatcher code, the documentation for that
code and the support for JSON job formats are now *deprecated*. As a
result, the corresponding lava-server code has also been *deprecated*,
this includes Bundles, Filters and Image Reports (original and 2.0).
The deprecated Dashboard is replaced by Results with Queries and
Charts which have been ported to the data produced by the pipeline
testjobs.
Future development within LAVA will be based solely on the pipeline
dispatcher and associated server side objects. Support for the
deprecated dispatcher will be maintained for future releases only
during the migration and *will be removed* once the migration is
complete. The migration is expected to take most of 2016. Individual
devices and particular items of job support will be gradually disabled
until the migration completes for all devices and all jobs.
The migration process involves:
0: Adding pipeline support for devices and deployment methods
1: Migrating user submissions and automated submissions to the
pipeline support. For the Linaro LAVA instances, this will happen
within the particular teams inside Linaro.
2: Making selected devices exclusive to the pipeline so that JSON jobs
can no longer be submitted for those devices. At this point, support
for JSON jobs which rely on those devices will cease.
3: Once all devices are exclusive, reject all JSON submissions.
4: Disable the old scheduler daemon on the LAVA instance.
At some point, probably in 2017 - a month or two after JSON
submissions cease - the migration will complete by executing:
5: Removal of code support for the old dispatcher in lava-dispatcher
and lava-tool codebases and associated documentation.
6: Removal of code support for database objects specific to the old
dispatcher, like Bundle, in lava-server with associated documentation.
This will involve the deletion of the Bundle data as well as image
reports, filters and BundleStreams. TestJob objects which are not
pipeline jobs will also be deleted.
7: Removal of the rest of the code which is still dependent upon or
only used to support deprecated objects and functions.
8: Modification of code which only exists to isolate the deprecated
objects from the pipeline objects. e.g. newly created jobs or devices
will default to pipeline support.
LAVA instances outside Linaro will need to manage their own migrations
during 2016 if updates are to be applied during 2017.
It is not possible to retain the database objects without the
deprecated code, so owners of individual instances may choose to
create an archive instance which has no online devices, accepts no
submissions, just provides access to a snapshot of the data at the
time that JSON submissions ceased. To maintain access to the archived
data, this instance must not upgrade to LAVA releases after the
archive is created and the original devices should persist in the
database but be kept in the Offline state. Devices should also have
submissions restricted to a single administrator account for which no
API token exists.
More information on the details of the migration and the state of
support for jobs using the old dispatcher will be announced on this
mailing list.
As a reminder - lava-announce is a read-only list. Posts to this list
are only made by the LAVA software team. Replies need to be directed
to
linaro-validation(a)lists.linaro.org - your email client should do this
for you, using the Reply-To header added by mailman.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Please reply to linaro-validation(a)lists.linaro.org
Please let us know if you are using OpenID authentication with LAVA.
Newer versions of django will make it impossible to support
django-openid-auth in Debian unstable and testing. The version of
django-openid-auth in Jessie can continue to be used, so we would like
to know how many users want to continue with this support.
OpenID as a protocol has been dying for some time and Linaro has moved
over to LDAP, which is fine if LDAP is already available.
The time pressure for this change is coming from the schedule to get
the latest django and the latest lava packages into Ubuntu Xenial
16.04LTS which means that support needs to be implemented in the
2015.12 or 2016.1 LAVA releases. This is why this is quickly following
the trusty change. We have been aware of the issues with
django-openid-auth for some time, it was only when we had completed
the move of the Cambridge lab to LDAP that changes involving
django-openid-auth could be considered.
If you are using OpenID authentication (e.g. using Launchpad or Google
OpenID), please let us know.
If you would like to see some other forms of authentication supported,
also let us know. We can investigate Python Social Auth
(http://psa.matiasaguirre.net/), if there is interest.
If we don't hear from users who want django-openid-auth support for
use on Debian Jessie, we will drop django-openid-auth support from all
lava builds. This will leave LDAP and local Django accounts in
2015.12.
If anyone has experience of other django authentication modules, also
let us know.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
See also:
https://lists.linaro.org/pipermail/lava-announce/2015-November/000002.html
which was also sent to these lists (except linaro-dev).
So far, nobody has come forward as a Trusty user. The only Trusty
instance of which we are aware is already due to migrate to Debian
Jessie.
The LAVA software team are now applying updates which will freeze LAVA
software support for Ubuntu Trusty at 2015.9 for lava-dispatcher and
2015.9.post1 for lava-server due to the complexities of supporting
both django1.6 and the current django1.7 in Jessie and django1.8,
possibly django1.9 by the time Debian Stretch is released.
The last packages for Ubuntu Trusty 14.04LTS will be:
lava-server 2015.9.post1
lava-dispatcher 2015.9
Once these changes are applied, the Debian packaging used to build
future versions of LAVA packages will prevent builds against django1.6
and prevent installation if django1.6 is found, in order to prevent
database corruption.
This means that Trusty users will not be able to use the results of
the dispatcher refactoring.
Ubuntu Xenial Xerus - which is planned to be the 16.04LTS in April
2016 - is expected to pick up LAVA software releases from Debian up
until the 2016.1 release (possibly 2016.2) and is also expected to be
using django1.8. The next Debian stable release (Stretch), for which
no date has yet been set, may use django1.9.
Initial attempts at migrating a test instance from Trusty to django1.7
did not go well and the migration from Trusty to Xenial cannot be
supported by the LAVA software team - the recommendation is to go
directly from 2015.9 on Trusty to the same version available for
Debian Jessie but there will still be work to be done to prepare and
implement the migration which will be instance-dependent.
Documentation is being added to assist with this migration but there
will remain risks of data loss which will need to be managed for each
instance. It is imperative that anyone using Trusty has an up to date
backup of the postgresql database dump before considering any
migration. If the existing data is to be dropped, a new install on
Debian Jessie is recommended.
It is not possible for the LAVA software team to support all versions
of django from 1.6 to 1.9 - particular problems are known when going
from django1.6 to django1.7 as the methods to migrate the lava-server
database changed fundamentally in django1.7.
Notes are being added to the documentation on the trusty branch based
on 2015.9 to be released within lava-server 2015.9.post1 and to the
documentation in the master branch (which will go into 2015.12).
All future builds of LAVA software will now be made and uploaded only
to Debian and releases.linaro.org.
So far, nobody has come forward who is willing to maintain packaging
for LAVA software on any distribution other than Debian. As the
refactoring proceeds, we expect that it will become easier to package
LAVA for other distributions but the migration to the refactoring must
be complete first.
Everyone interested in or using LAVA is encouraged to subscribe to the
lava-announce mailing list which is low volume and only used for
substantial changes like this.
https://lists.linaro.org/mailman/listinfo/lava-announce
See also https://validation.linaro.org/static/docs/support.html
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
This is a call to *all* users of LAVA on Trusty - please let us know
who you are, what you're needs are and whether you are able to migrate
to Debian Jessie instead of going from Trusty 14.04LTS to Xenial Xerus
16.04LTS.
If you are using LAVA on any release of Ubuntu older than Trusty, the
only advice from the LAVA team is to immediately migrate to Debian
Jessie.
If you are using LAVA on Wily Werewolf or Vivid Vervet, you should
also consider testing the migration to Xenial and comparing with a
migration to Debian Jessie, as below.
LAVA is looking at a fix for the 2015.9 release but Django1.9 is in
beta release already. There are issues with django1.8 which are in
development. Currently, we are proposing that this update will be
applied to 2015.9 and made into a frozen release.
The master branch of LAVA will continue to develop and will need to
use more features only available in Django1.7 and later. Documentation
relating to installing Trusty would then be deprecated and removed in
subsequent releases from the master branch. Equally, future database
migrations on the master branch would no longer provide south support,
using the django migration support provided by django1.7 and later. So
these releases would not be built for Trusty - leaving only the frozen
branch.
Due to the complexity of supporting django1.6, it is unlikely that
updates will be available for the frozen branch once this happens..
The changes in the lava-server due to the ongoing refactoring will
mean that users of Trusty will be unable to migrate to pipeline
support until the server has also been migrated to Xenial 16.04LTS.
In addition to this, there is concern that migrating from Trusty and
django1.6 all the way to django1.8 or possibly django1.9 in Ubuntu
Xenial 16.04LTS is going to be problematic and the LAVA team will be
unable to assist in most cases.
The alternative is for someone with a reasonably complex lab running
Ubuntu to take up a role as tester of the frozen branch *and*
responsibility for patches which can maintain trusty support and
migration to Xenial 16.04LTS. The problem then will be that it will be
a very large transition when 16.04LTS actually becomes available -
only for the same lag to start all over again.
I'm unsure when Xenial will close the window for migrations from
Debian into Xenial - I expect that the 2015.12 release of LAVA will
migrate, I expect that 2016.1 will migrate too but I cannot be sure
about 2016.2 or 2016.3. That migration is completely outside the
control of the LAVA software team.
https://launchpad.net/ubuntu/xenial/+source/lava-server
Everyone considering staying on Ubuntu is advised to try a migration
to Xenial *now* - in a VM, with and without a recent backup of your
database and logs. Xenial currently has 2015.11. Also compare with a
migration to Debian Jessie by dumping and reimporting the database. In
each case, ensure that the permissions on /var/lib/lava-server and
sub-directories are retained from the original.
Please talk to us and test out what you are going to do.
https://validation.linaro.org/static/docs/support.html
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
It has become impractical to maintain support for the latest production
releases on Ubuntu Trusty 14.04LTS due to behaviour required by the
server side changes needed by the dispatcher refactoring. These are
important changes and are impractical to backport to the version of
python-django available in Trusty.
I have tried working around the issues but it involves pulling more
dependencies from Jessie or Vivid and in my tests, the existing
installation became confused and failed to install due to problems with
the database migrations. The LAVA software team do not have the
resources to maintain backports of the complete dependency chain in
Ubuntu, it is a much more manageable set in Debian Jessie (where the
same migration went without issues on multiple machines).
The changes in 2015.11 are almost exclusively related to the dispatcher
refactoring and therefore there is expected to be limited impact on
current installations on Trusty.
For the next release, we will investigate code changes which omit the
new support for Trusty builds but this will have the effect of
preventing those installs from working with the pipeline and the new
YAML job submissions. The documentation for Trusty will also be
updated, if necessary as a hotfix to 2015.9 just for Trusty.
If the code changes do not work cleanly, then the current 2015.9 (and
the documentation hotfix to follow) would become the last LAVA releases
for Ubuntu Trusty 14.04LTS.
Users of Ubuntu Trusty 14.04LTS are reminded that a new LTS is due for
release in April 2016 but that the LAVA software team are not using
Trusty any longer. Any issues with the migration from Trusty to the
new LTS are outside of our control.
Migrating an instance from Ubuntu to Debian would involve dumping and
re-importing the database - bear in mind that Trusty has postgres9.3 and
Jessie has 9.4, so care will be needed to migrate the postgresql
cluster correctly - there is documentation for that migration in the
lava-server docs. It is not recommended to attempt to migrate the OS
itself from Ubuntu to Debian (or vice-versa), a reinstall is a better
option, so make a backup and ensure that the backup works.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
After discussions at the LAVA Users Forum at Connect SFO15 [0], the LAVA
team have arranged for some changes in the mailing lists to make it easier
to announce and manage the upcoming changes from the ongoing refactoring.
https://lists.linaro.org/mailman/listinfo/lava-users - a discussion list
amongst users for users, with some admins listening in and helping out.
Users in this sense includes test writers and administrators of LAVA
instances outside Linaro. It is particularly aimed at those running LAVA in
places where Linaro is not necessarily aware of the install. I'm not asking
for LAVA instances to declare themselves, it is free software after all, it
is an offer of assistance and a request that users take advantage of the
support available to manage the implications of the refactoring for their
own use cases. Please feel free to initiate threads and discuss issues and
problems on the lava-users list.
As a reminder, for those who use LAVA but have not recently been to a
Connect, the dispatcher is being refactored and this had led to
advancements and modifications in the lava-server as well as a completely
re-written job submission format. LAVA is retaining compatibility *only*
with the Lava Test Shell Definitions (the YAML files people are currently
using) and there can be no automated way of converting existing JSON job
submissions to the new job submission format (which uses YAML to allow for
comments, amongst other improvements).
The timetable for these changes is expected to cover most of 2016. After
https://validation.linaro.org/ has migrated to the new code, the current
dispatcher code will start to be removed from the upstream in 2017.
One result of these changes is that as the Linaro production instance
migrates away from the current dispatcher codebase, there is a likelihood
of some breakage in some parts of the current dispatcher. (This is one of
the primary reasons why the dispatcher is being refactored.)
The refactoring introduces a lot of benefits, including much more robust
communication between the workers and the master, removal of configuration
on the workers so that admins only change things in one place, a lot of new
methods within the dispatcher to support new types of test and a much
cleaner, more modular, codebase for future development.
Also, the Linaro Validation Mailman List <linaro-validation(a)lists.linaro.org>
is to be renamed (keeping linaro-validation as an alias) to lava-devel for
consistency with lava-users and lava-announce. There will be a separate
email to the linaro-validation list about these changes shortly.
These new mailing lists will be included in the documentation included in
next release of the LAVA software (2015.10). Documentation of the new code
is already in the master branch of lava-server and installed via the
lava-server-doc package and the initial code for the refactoring is being
developed in parallel with the current code. This list will be used to
announce when the refactoring codebase is ready for admins to begin
migrating tests and test writers to the new code and methods - currently,
the new dispatcher is developer only. At SFO15, there was also an update on
the state of the refactoring. [1]
lava-announce will continue to be a low volume list. Reply-To has been set
to the linaro-validation mailing list (transparently migrating to
lava-devel once the linaro-validation alias is in place).
The LAVA developers would ask that everyone running LAVA would subscribe to
at least the lava-announce mailing list, to help with the migration to the
new support.
Videos will be available for most meetings at SFO15 in due course. The
slides are already available.
[0] https://sfo15.pathable.com/meetings/302656
[1] https://sfo15.pathable.com/meetings/303074
--
Neil Williams
=============
Tech Lead, LAVA software team.
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
