As with previous releases since 2016.12, the 2017.6 release is only
available from the lava repositories:
https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…
However, Debian Stretch is due to be released very soon. 2017.6 may be
the first release to be uploaded to Debian unstable since the stretch
freeze started with 2016.12. There may be a short delay whilst
unstable settles down after the usual round of extra uploads. Once
packages migrate into the new testing (buster) and stretch-backports
becomes available, packages will be uploaded for stretch-backports and
jessie-backports.
Device dictionaries moved to the filesystem
===========================================
LAVA-757 - all existing device dictionaries will be migrated to the
filesystem (/etc/lava-server/dispatcher-config/devices/) during the
installation of the 2017.6 production release. The dependency on
python-django-kvstore has been dropped. This means that there is no
longer a need to update the device dictionary in the database, simply
install the new file into the directory and the next test job to be
assigned to that device will use the new configuration. Unlike
device-types, there are no default device dictionaries provided by
LAVA, so a fresh install will have no device dictionary files in
/etc/lava-server/dispatcher-config/devices/
The documentation has been updated. There is now no need for
lava-server manage device-dictionary, so this command has been
dropped. Instead, admins who have permissions to run lava-server
manage can simply copy a device-dictionary .jinja2 file into
/etc/lava-server/dispatcher-config/devices
During installation, this migration will take place:
Applying lava_scheduler_app.0027_device_dict_onto_filesystem
Database entries for all device dictionaries are replaced with files.
If a device can support V2 but has no device dictionary, it will be
marked as Skipped.
This change means that lava-server no longer depends on the
python-django-kvstore package.
Feedback support with namespaces
================================
Devices may output messages to the serial console as a result of
operations performed within an LXC and this output is now logged as
part of the test job log file as 'feedback'. See the documentation for
more information on feedback output.
Architecture is optional for LXC
================================
Some LXC deployments support identifying the deployment architecture
based on the host system. Some instances may use workers of different
architectures and test jobs should be portable between instances.
Therefore, the arch: field of the lava-lxc protocol has been made
optional. Note that Fedora and Ubuntu deployments still need the
architecture to be specified for x86 machines to distinguish between
32bit and 64bit x86 deployments.
The architecture value can still be specified in test jobs but the
test jobs will be more portable if omitted.
Action names use hyphens, not underscore
========================================
If you are using action-specific timeouts in your test jobs or in
customised device-type templates, note that the names of some actions
have *changed*. This is to make the usage of hyphens and underscores
consistent. Actions with names containing underscores are now invalid
and all underscores in names have been replaced with hyphens.
'pdu_reboot' becomes 'pdu-reboot'.
lava-server changes
===================
daf1e2a lava-master: fix init script
b6b5fe7 Reduce device configuration dynamic connections
ebf4463 Fix a crash in the migration when the db is empty
cdfeb12 Fix index out of range exception
33092ef Adding usb deployment method to vexpress device template
3257ee2 XML-RPC: remove unused imports
55dd014 XML-RPC: fix crash when called by anonymous users
a3d4718 XML-RPC: allow every user to get templates/dict
ddaf693 Add documentation of feedback support.
eaae89a Set a default connection timeout for lava-test-shell
f305e47 publisher: fix umask
c25686f logrotate: lava-master should be owned by lavaserver
2da8e74 logs: except more exceptions
14f5f6c Update docs for changes in 2017.6
a486d12 Tidy up of debian support docs
b05712d Preserve original comments when resubmitting
19bd2db XML-RPC: add an api to manage aliases
30973a0 Log an error when saving device configuration fails
763239d Use --arch for fedora LXC example
8f26102 XML-RPC: add a method to show tag details
19a69f7 fix location of example test job files
480a6f6 XML-RPC: add an api for handling jobs
356667d Update installation docs for Stretch release
da74ae5 XML-RPC: add a decorator for is_superuser check
49f370c XML-RPC: add a clean api for device-types and aliases
c6d503d commands/devices: extend devices command to add tags
1655dbf Move set commands outside body block
87b8c01 XML-RPC: add a clean api for workers and tags
351c16b XML-RPC: add a clean api for devices
c4dec1f device-types: add bcm2837-rpi-3-b
9ca0e0d Adding combined vexpress device template
67f531f Make lava_scheduler_app.api a module
d3021a3 xmlrpc: add a system.version function
3ef0157 linaro_django_xmlrpc: remove unused properties
e994852 Use render shortcut instead of the loader
e51ca21 linaro_django_xmlrpc: rework the application
4bbe17c LAVA-959 run lava-master as lavaserver user
b283288 Fix migrations from Debian Jessie to Stretch and ensure smooth upgrade.
bc25204 LAVA 329 - Add old job URL to metadata when resubmitting a job.
271723d LAVA-782 - Change error type for incorrect handling of custom queries.
368903e notify: Don't send verbose mails when the job is cancelled
71c514d Fix broken link in unit test.
1f48674 LAVA-955 - Remove 'arch' parameter from lxc protocol.
3b36ba0 Add auto_login test case in test_pipeline unit test
e23bb17 Fix typo which causes XML-RPC error when saving device dict
9a82aee XML-RPC: allow to have sub_modules
b0c7a5b Document reboot and apply-overlay elements for fastboot deploy method.
8a93049 LAVA-935 - Provide feedback output from connections
d71f267 Fix typos in proxy configuration documentation.
c7227b3 Fix rebase error in grub hikey test
9253497 Only the master should create log directories
1ee736b Allow every u-boot devices to use minimal boot
e816c17 LAVA-832 allow to manage users from the command line
eb83cbe Add grub interrupt prompt and interrupt character to constants
ee79de2 LAVA-937 show job visibility and fix group visibility
73b6d86 Prevent key error in metadata handling.
f89e840 Improve job output migration helper
eb1ad02 device dict: do not show exclusive flag
3900276 Restore date-based subdirectories
1faa9f0 LAVA-855 - ART CI: Intel NUC device integration for LAVA v2
d195b01 Allow to revert (dummy) the migration
35f0802 Fix device dict export
da7ebe2 device dict: fix web rendering
4a3daa9 Allow override of root for installed mustang.
12c0508 Expand zmq_client to handle publisher socket
34f3625 Trivial whitespace changes in glossary
6246a5d Improve data export docs
2af8e72 Allow for non-Const values in device dictionary
59742fe Extend examples to use the DNS support for NFS
c514296 Fix change in import paths for V1 and V2 scheduling
765901e Add notes on advanced features for submission
fd0905c Remove deprecated commands
0237b85 Use CommandError whenever possible
a1cc08b Improve device-type templates
05123cc Missing conditional in device-type alias handling
17af14e Move commands to lava_server app
5bab54d Remove reference to unavailable boot method 'fastboot-boot-image'.
78e9048 Add mediatek dt for v2
49f27fa Allow in-place YAML include in V2 job submission.
c2be83f Grub support for HiKey
ae4d477 LAVA-757 Move device dictionaries to file system
52eb1e1 Remove unused model JobPipeline
9d0a5ae results: make the link unique
55bf39f Add a callback_url functionality to notification schema.
0324915 Add login_commands to the auto_login schema
ce6d0f7 Don't reload template from disk on each access
lava-dispatcher changes
=======================
38da5d8 LAVA-974: Fix init script and allow to set hostname
0fd687e Don't use os.path.realpath() on device names
d84dab7 Modifying vemsd deploy action to use ResetDevice
d21385e testdef: get 'revision' from the right variable
5ea16fe Rework test shell feedback timeouts
510c4e5 Declare the feedback timeout
b994757 Use pyudev device links as well as device nodes
e609c22 Fix namespace usage in power commands.
39e2e7e Read from feedback connection during long operations
7be926a Allow tar.gz firmware bundles
20e66c1 overlay: Raise JobError if root_partition is None
4e27fcd LAVA-935 - Provide feedback output from connections
831c71c Fix bug #3032 - Running an LXC/Hikey job without a test action ...
84e7378 Fix namespace issue in NFS deployment action
f1f8d71 Add login_prompt valid test case
7dc48aa Issue lxc-stop before destroying a container.
3e1cb07 Changing udev checks to only look for FS label
d08db39 Fix #3010: set logging timezone to UTC
1b664e5 LAVA-955 - Remove 'arch' parameter from lxc protocol.
8a4dbc7 Increase LXC finalize timeout to 90 seconds similar to multinode.
3823c49 Use one debug log call to log the same kind of message in lxc protocol.
5db313c Creating vemsd mount points
ea03ef1 Fixing boot test issues using NFS deploy action
d1f4a30 Fix rebase error with grub prompt constant
b2aef8b device-types: add sun6i-a31-app4-evb1
8ceca07 Make grub use interrupt prompt and character from constants
b495a35 device-types: add ttyAT0 to Atmel devices to support multiv7_defconfig
3a89027 device-types: fix at91sam9x25ek to use bootz
9e00895 Set download max_retries in the constructor
959480c Add minimal boot action
31fd021 User command: do not clean if it was not executed
26572a2 Fix keys and label that were outdated
67dd60e Remove unused functions
0c100ff Remove boot-result common data
342039a Fix RetryAction inside a RetryAction
9ce430f Fix check on LXC for default usage.
0576754 Fix typo: Use pixel job create function for testing pixel device.
e7a4422 Fix typo: remove visibility added twice.
9e67b6d LAVA-855 - ART CI: Intel NUC device integration for LAVA v2
dd5d874 Ensure all deployments can add LXC devices except QEMU
e72d490 Fix pipeline refs for change in monitor retry
cb57d2b LAVA-942 UEFI Menu and command lists
e57013b Start adb daemon before attempting adb commands.
ac389ad lava-slave: remove tmp dir when END_OK is received
632a0ff Fix device configuration
0cffce4 Fix validation of boot auto_login parameters and add a unit test
7e40701 Allow unix line separators for some UEFI devices
095798a Avoid calling cleanup in validate
f98dade Allow updating all pipeline_refs at a time.
31b62dd LAVA-928 - HiKey issue when switching between AOSP and OE
51f40ef Make soft reboot optional LAVA-846
4d43d02 HiKey Grub EFI support
874f889 Avoid logging the same message twice.
3be27df lxc protocol: simplify the tests
75126ff Do not set self.errors when Action.run fail
ce32bfb Do not use self.errors in Action.run()
a5d9eba Fix detection of missing ssh_host value in validate
234eef9 Fix calling of protocols after LXC change
477a36b tests: make ShellCommand.logger a DummyLogger
62867f4 Use images.validation.linaro.org files for unit tests
1868936 Silence logging in more unit tests
609c783 Fix download action name
b170197 More tweaks to silence messages from the unit tests
29efa26 Adjust for pep8 checks in jessie
0733f89 Make sure the test_character_delay is used for all commands in
test shell
78d2509 Account for empty environment string.
ea09b19 Take namespace into account when counting test stages
734688c Declare namespace of the test suite in results
60561c2 Parser: remove unused context
49a6b92 Drop noisy info log message.
8ed8467 lava-test-monitor: use TestMonitorRetry action
4fd5d4a Drop unused imports and unused variables
678b216 Rework the removable action to allow sd cards
3b42122 Fix bug #2975 after qemu-nfs introduction
7462ea6 Expand LXC support to add devices from all sybsystems
b853865 Fix missing check for u-boot commands parameters
8293693 Replace invalid characters in test_case_id.
a94e0f2 Fix preseed/late_command appending
9647d55 Run login_commands if provided in auto_login
db03483 Fix Action names (use - instead of _)
9e836e6 Add a Command Action
bbd4ba2 Make the parser stricter about the block names
65b3a56 Allow to pass integers to run_command
b41326b Raise JobError instead of NotImplementedError
073f3b3 update gitignore
00f5d33 log: limit the length of lines send other zmq
3f99cc3 Add missing super call to WaitUSBDeviceAction
5b2b074 LAVA-889 Fix handling of multiple test blocks
5436658 Fix indent typo
3bfdcdb Add support for use_xip
8df17dd Add support for append_dtb
3c110ca Remove unused imports
4847bf1 Put device rebooting info message at the correct place.
6b85382 Fix call to lava-lxc protocol for pre-os-command
176a4d4 Add new utility function infrastructure_error_multi_paths
bdb4d12 Fix test_lxc_api unit test
dc008eb Export the full version string.
e9df320 LAVA-920 - Workaround ptable issue in firmware
7b4096c Allow LXC protocol to call pre-os-command
b33d04b Fix OE image support for HiKey.
d714a9b Add missing test shell helpers
daebfad Extend secondary connection fix to support primary
9518e74 Add missing import
e97801d Add optional deployment complete message list to secondary
media deployments
fae0ced Move unnecessary constants into base jinja template.
18f745b device-types: add kirkwood-db-88f6282.conf
815e518 device-types: add at91sam9m10g45ek.conf
6ca0731 device-types: add at91rm9200ek.conf
003b53d device-types: add at91-sama5d4_xplained.conf
9b2371c device-types: add armada-3720-db.conf
5a76733 device-types: add armada-xp-gp.conf
8b05093 device-types: add kirkwood-openblocks_a7.conf
f5801d3 device-types: add alpine-v2-evp.conf
19e2027 device-types: add sama5d34ek.conf
49da704 device-types: add armada-385-db-ap.conf
ed7aa6b device-types: add armada-370-db.conf
9a10432 device-types: add at91-sama5d2_xplained.conf
ddb34ce device-types: add alpine-db.conf
f7642f5 device-types: add armada-375-db.conf
0d70c7d device-types: add at91sam9x35ek.conf
a642d45 device-types: add armada-xp-db.conf
2a5ff9e device-types: add armada-7040-db.conf
1a43999 device-types: add orion5x-rd88f5182-nas.conf
ad855a6 device-types: add armada-xp-linksys-mamba.conf
58308ca device-types: add armada-388-gp.conf
6a2898b device-types: add armada-370-rd.conf
cd1b4f4 device-types: add armada-398-db.conf
51925fb device-types: add sun8i-a83t-allwinner-h8homlet-v2.conf
4976d06 device-types: add sun8i-a33-sinlinx-sina33.conf
681480d device-types: add sama5d35ek.conf
db74d60 device-types: add sun5i-r8-chip.conf
40736b9 device-types: add sama5d36ek.conf
433be0a device-types: add imx6q-nitrogen6x.conf
ed4c82e device-types: add at91sam9x25ek.conf
ce7a488 device-types: add at91sam9261ek.conf
f87cdb4 device-types: add armada-xp-openblocks-ax3-4.conf
634442c device-types: add armada-388-clearfog.conf
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
This is a long mail but there is a lot to cover...
Stretch
======
Debian Stretch is expected to be released on June 17 2017:
https://lists.debian.org/debian-devel-announce/2017/05/msg00002.html
As has been mentioned on this list before - admins MUST check the apt
sources for all machines *NOW* to ensure that all your Debian apt
sources are pointing at 'jessie' NOT 'stable' if you are to be able to
manage the upgrade yourself.
What you want in /etc/apt/sources.list and /etc/apt/sources.list.d/* is:
deb http://mirror.bytemark.co.uk/debian jessie main
deb http://mirror.bytemark.co.uk/debian jessie-backports main
(The mirror you choose is up to you.)
If you find any apt sources referencing 'stable' or 'testing' CHANGE
THESE NOW or those systems could automatically upgrade to Stretch or
Buster respectively on 17th June.
As with all such releases, production machines are recommended to
allow some time for final testing before doing the upgrade. The
Cambridge LAVA lab is expected to upgrade from Jessie to Stretch
during August 2017. The LAVA software team have been doing testing of
this upgrade for some time, without issues. However, real machines
with extra packages and local admin configuration cannot be tested in
this way, so each lab should take some time to plan when and how to do
the upgrade. (Hint: have a backup and check that it can be restored
correctly before starting.)
https://validation.linaro.org/static/docs/v2/admin-backups.html
There has been a fix for lava-server in stretch to smooth the upgrade
for those users who have not installed lava-server from
jessie-backports. Those already running 2017.* are not affected by
this change and can continue using jessie-backports and the LAVA
repositories as before. Stretch will release with lava-server
2016.12-2 and python-django 1:1.10.7-2
Once Stretch is released, Debian testing will be called 'buster',
Debian stable will be 'stretch' and Debian oldstable will be 'jessie'.
At a later date, jessie-backports will be opened again and a new
stretch-backports will become available. New LAVA production releases
will be uploaded to unstable to migrate into testing (buster) and then
be backported to stretch-backports and jessie-backports. The LAVA
repositories will be updated as well.
Support for jessie-backports will continue after stretch-backports
support is added but cannot continue indefinitely. At some point, all
instances will need to have moved to Stretch to continue getting
updates of LAVA. (Prior to the process of releasing Debian Buster, the
jessie packages will be dropped from the archive, including
jessie-backports.)
Changes in LAVA repositories
=======================
2017.6 will be released in the same way as previous releases in 2017 -
to the LAVA repository for use on jessie using jessie-backports.
At a later point, 2017.6 will be uploaded to Debian unstable for
migration into testing (buster). Only developers should install that
version as Debian unstable can be problematic in the first few weeks
after a Debian stable release.
There are changes in how the lava packages are built on jessie versus
on stretch, so the lava repositories will be split. I'll be updating
the documentation in 2017.6 to cover the changes required to help
admins use the new suites. There will be a new jessie-backports suite
and a new stretch-backports suite. The existing 'sid' or unstable
suite will have no uploads after 2017.6.
This will be covered again in the 2017.6 release email and again with
2017.7. For now, no changes are required.
Once the new suites are announced, admins who choose to upgrade to
Stretch would use the stretch-backports suite from the LAVA
repository.
deb https://images.validation.linaro.org/production-repo stretch-backports main
Those who will upgrade later, would use jessie-backports from the LAVA
repository.
deb https://images.validation.linaro.org/production-repo jessie-backports main
The purpose of these suites is to allow admins to choose to upgrade to
the latest production release for jessie or stretch without waiting
for the migration of the packages into Debian testing and the
subsequent uploads to backports. The packages are otherwise the same.
Django
======
Longer term, there have been a lot of discussions within Debian about
managing django releases. The plan is that only django LTS releases
will be available in Debian unstable, testing and therefore stable
after the Stretch release. Non-LTS releases will only be available in
Debian experimental. This means that lava-server will migrate from one
django LTS to the next django LTS with future Debian releases. The
LAVA software team will manage how deprecation cycles within django
affect lava-server.
This does have the issue that the version of django in Debian stable
will be towards the end of the upstream support cycle but it is better
than having a non LTS django release in Debian stable. Synchronising
releases across multiple upstream teams is not feasible.
The next django LTS release after 1.8 is 1.11 which is expected to be
in the next Debian stable release - buster.
https://www.djangoproject.com/download/
Python3
========
The last django LTS release to support python2.7 will be 1.11 LTS
which retains upstream security and data loss bug fix support until at
least April 2020.
The LAVA software team have already been testing lava-dispatcher V2
with python3. Once the V1 code is removed from lava-server, testing
will proceed for lava-server too. We fully expect to be able to
support lava-dispatcher V2 and lava-server V2 on python2.7 and python3
for a limited time.
There will never be python3 support in lava-dispatcher V1 or lava-server V1.
Python2.7 support for lava-dispatcher V2 and lava-server V2 will be
removed at some point prior to the release of the next django LTS
after 1.11 which is 2.2 LTS, due in April 2019.
Once django 2.0 (non-LTS) arrives in Debian experimental, the LAVA
software team will begin testing lava-dispatcher V2 and lava-server V2
with python3. For some period in 2018, both python2 and python3 will
be supportable but only when building local developer packages. LAVA
production releases will stay on python2.7 during testing with
python3. Once we are happy that python3 support is ready, there will
be an email to this list and the following production release will
provide only python3 packages. This change will be transparent for
most instances, the upgrade will simply pull in a lot of python3
packages and the python2 packages will be left unused.
Reviews and contributions to lava-dispatcher V2 already run the unit
tests with python3 and once the V1 code is removed, this will start to
happen with the lava-server code as well.
LAVA can move to python3 at any time once the V1 code is removed - all
the necessary support is already available in Jessie. LAVA will have
to move to python3 before django 2.2LTS is the only version of django
released in Debian stable.
Details of how this will be managed are yet to be defined. We expect
to continue parallel support of python2.7 and python3 for a limited
time.
It is likely that support for jessie-backports would need to cease
before support for both python2.7 and python3 is included, just to
manage to matrix of builds and tests. LAVA can support python3 using
django 1.11LTS but must drop support for python2.7 to support django
2.2LTS.
Summary
=======
17th June 2017 - expected date of Debian Stretch release
August 2017 - migration of Cambridge lab to Debian Stretch instead of Jessie
November/December 2017 - removal of V1 codebase
Q1 2018 - testing with django 1.11 LTS
Q2 2018 - testing with python3, likely end of support for jessie-backports
Q3/4 2018 - add python3 support
Q1 2019 - drop python2.7 support.
April 2019 - release of django 2.2LTS
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
