Roadmap reminder
================
https://lists.linaro.org/pipermail/lava-announce/2017-September/000037.html
Installing 2018.1 will apply the changes from 2017.12 and earlier to
**permanently erase V1 test data** if this has not been done already.
(i.e. if you skipped installing 2017.11 or 2017.12, remember that 2018.1
includes the changes of 2017.11 and 2017.12.)
Removal of V1 test jobs
=======================
A few provisos with this release - there are a lot more changes to the
lava-server package than to the dispatcher because 2018.1 drops the old
lava_scheduler_daemon app and moves all scheduling into lava-master.
The new scheduler state machine involves some database migrations:
Applying
lava_scheduler_app.0032_add_state_health_to_testjob_and_device... OK
Applying lava_scheduler_app.0033_remove_old_status_fields... OK
Applying lava_scheduler_app.0034_worker_last_ping... OK
If your instance has a lot of V1 test jobs still, the migration could
timeout, so as the final part of the removal of V1, follow the guidance in
the documentation to remove V1 test jobs, if you have not already. (By "a
lot", we're thinking more than a few thousand.)
https://validation.linaro.org/static/docs/v2/v1removal.html#removing-v1-fil…
If you have millions of V1 test jobs (as validation.linaro.org had), you
will need to stagger the removal of the V1 test jobs using the --older-than
support. Talk to us if you have issues, there is an updated script which we
used on validation.linaro.org but that would need to be locally
cherry-picked from 2018.1 onto the system running 2017.12.
The removal script can be interrupted with Ctrl-C and restarted if
required. Depending on the performance of the host machine, it may take a
long time to run the removal, up to several hours in some cases.
Finally on the removal of V1 test jobs, the support removes not just the
database object but also the log file directory, so your master should find
a lot more usable fileystem space has appeared, once complete. In many
cases, the log file directory can be missing for a V1 test job but the
warning is just for information. The database object will have been removed.
Removal of unwanted dependencies
================================
$ sudo apt-get --purge autoremove
Do check the list of packages before proceeding. If there are packages
which you need for local support, make sure those packages are marked as
"manually installed" in apt by using apt-get install <package_name> (this
doesn't reinstall an existing package, it simply marks it as not a
candidate for autoremove).
It is worth running autoremove after this release has been deployed, extra
dependencies can still be a security problem and an admin burden.
Stretch
=======
2018.1 *deprecates* Debian Jessie and moves the recommendation on to Debian
Stretch as the supported OS for LAVA.
https://lists.linaro.org/pipermail/lava-announce/2018-January/000046.html
Note that when the instance is upgraded to Stretch, this will bring in a
new major version of PostgresQL and your LAVA database will need to be
*manually* migrated. Neither PostgresQL nor Debian will do this for you.
https://validation.linaro.org/static/docs/v2/debian.html#migrating-postgres…
When following this guide, remember that your versions will differ from the
guide - typically for the upgrade to Stretch, you will be migrating from
9.4 to 9.6. Take care with the commands to ensure you drop and upgrade the
correct cluster. Make sure you have a valid backup. The instance will need
to be in a *maintenance window* for the migration to occur.
Repositories
============
2018.1 has been uploaded to stretch-backports and jessie-backports for the
production-repo on images.validation.linaro.org and jessie-backports will
continue to receive updates until we get closer to the end of security
support for Jessie. See
https://lists.linaro.org/pipermail/lava-announce/2018-January/000046.html
lava-server
===========
d87ccc73 Restore Health and TestJob Reports calculations
6aa325e0 Add a reason when updating device and worker health
aaa1a959 Notifications: use email signature conventions
3f6770f4 Fix notifications and log the exception
fbe57d5f Only use get_change_message on Django > 1.10
832edcf8 fix pep8 error
ff1ef1c4 Disable high limits on ODROID-XU3
7ae61e90 Remove old sample job
4be662da LAVA-1206 - Improve docs for API chunking
6c717e49 LAVA-1207 - Add xmlrpc API call for test suite list.
d8d7ffdd Fix pep8 - E127 continuation line over-indented for visual indent.
81a2a6d1 Return state and health information as strings not numbers.
d2451a1d LAVA-1163 - Unify json library across lava-server.
d9ea9343 LAVA-1192 document use of static_info
824d9580 Deprecate Jessie in favour of Stretch
0791d8be LAVA-1198 - Make sure API covers retrieving the set of test suites
489b83da Add optional support for python3 unit tests
91782fac A sequence is required before excluding a field in table.
6be77f09 AUTHENTICATION_BACKENDS list is available in distro settings only.
938352ff Add documentation for the depthcharge boot method
f6172120 Ensure added blocks are wrapped in conditionals
7ca1a522 LogEntry: improve SQL queries
38c8c8e3 Improve LogEntry tables
e6bb9cf3 LAVA-1199 - Implement incremental result exports in XMLRPC
2945e5f7 Simplify TestJob and Device state display
2e6460bd LAVA-1197 - Document REST API chunking on test suite level.
2f7bc9ab Fix pep8 error
4f896471 LAVA-1197 - Implement incremental result exports in REST API
7a0854ac LAVA-1103 document user and group creation
4b7b353a lava-master: fix string serialization
1e9c1aa4 LAVA-1119 - users and groups for devices
4963ffa9 LAVA-1102 - add group support to manage commands
bf9dd4ec Add missing sub-command parser for device update
9a480bc0 Fix missing import for lava-master
39faeb5f fix crash when canceling a non-scheduled job
4f201dd6 master: use yaml.dump to dump a dictionary
045ed9ec schema: validate multinode
2d2963d9 Allow browsing LogEntry in the admin interface
ba37046c fix pep8 error
8c6b3c35 LAVA-1186 - Fix python3 unit test support
be8b1e5d Worker: record the last ping received
1f4157c1 lava-logs: fix issue when unbinding under python3
d64514fb Port linaro_django_xmlrpc to python3
4b04cc30 lava-server manage: make the sub command required
f49a1bd7 Port scheduler.jobs api to python3
320cb918 jobs: split the database requests in small chunks
babab5ce Prevent noise from apache2 for DocumentRoot
79cfc375 Add FIT parameters to base-depthcharge and rk3288-veyron-jaq
62dd2464 python3: port timing page
b52c84c6 Port BreadCrumb to python3
c4432f55 lava-master: sleep a bit when db connection drop
6b71a7e0 Port lava-master to python3
7f0a2ae5 Improve support for python3
e2182b3c Port lava-logs to python3
cc05ce30 All images that accepts compression will also accept archive.
2bb99530 Port lava daemons common functions to python3
1e84fdc9 Remove deprecated settings
c530a82e remove restrictions on worker health transitions
f66b7f6b workers.show now returns the list of devices
36239079 Print LogEntry messages on the admin main page
02f83e49 Test the Device and TestJob state machine
2d427a84 Remove unused job json functions.
70cb792c LAVA-963 - Support python3 -Wall unit tests
34d52727 Add base-depthcharge and rk3288-veyron-jaq device types
60f85e3e Derive protocol elements from sample jobs.
012034ef Update app_index.html from last django version
849aaa6c manage jobs rm: try to be nice by sleeping a bit
57a2356d Allow for instant canceling
bc754b3f Fix udecode crash when receiving unicode strings
6aad013a Generate an event when a job is submitted
ed9546fc master: improve scheduling regularity
698817ae Fix fastboot boot sequence for hi960-hikey device type.
107cea4b Fix 500: remove_broken_string has been removed
7c95d706 Extend base-fastboot template for fastboot device types.
d779a194 Python 3 support for cStringIO.
65e769d7 Update example test job with pre command
2467411b LAVA-1170 - Unicode literals port
3bf73425 Provide and Document the storage_info support.
270c90a4 Document the state machine and scheduler
87af6a56 LAVA-1169 - Support xmlrpc.client imports in python3
e24865f3 Update lxc fedora example
14792136 No need to save the object returned by get_or_create
6114d0f9 Rewrite from scratch the job scheduler
9da8d83f Fix broken line detection
af632f12 Add a test character delay to rpi3
a5163e61 LAVA-962 - Port lava-server to python3 exclusively - urllib
b7e7ec2a LAVA-1085 - First time 'run query' click does not
enable 'Results' button
a47c170e LAVA-1084 - Clicking on 'live query' tooltip will submit request
e50c2d03 LAVA-1082 - Results action link points to the same page
6705ec87 Refresh the setting files
e4b39c38 Store the lava_server static file in the right dir
lava-dispatcher
===============
b0de4b84 Enabling energy probe data to be passed to LAVA test shell
dd349a77 Use BaseException only when needed
c1a0cd99 lava-slave: when canceling, wait for the job
f00a7315 Port multinode to python3
f937a861 docker deploy: fix python3 port
11e3571d Generate the FIT image for Depthcharge
d46f10c4 Update unit test sample file for images.v.l.o mirror.
b4536e09 Include child devices of matched pyudev devices
eb04f6fd Remove V1-only test shell helper scripts
9f96a0c5 utils/udev.py: lxc_udev_rule only use logging-url arg
when is not None
781312b5 devices/db410c-01.yaml: Add cdt after aboot in flash order
9e809927 LAVA-1172 add support for lava-target-storage
53cfc7db Add support for the depthcharge boot method
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
As announced in June 2017, Python2 is end of life.
https://lists.linaro.org/pipermail/lava-announce/2017-June/000032.html
However, the outline of the timeframe needs a few changes as Debian Jessie
will lose security support around June this year and all LAVA instances
will need to be running Stretch before that happens. It has proved to be
impractical to support LAVA using Python3 on Debian Jessie and the
completion of the migration has meant that lava-server is ready for Python3
ahead of schedule.
Therefore, the pending 2018.1 release will **deprecate Jessie** and move
the recommended installation of LAVA to Debian Stretch.
The 2018.2 release of LAVA will add Python3 support to support developers
and contributors using Python3.
This means that the 2018.2 LAVA packages will contain python code in both
the existing Python2 directories: /usr/lib/python2.7/dist-packages/ as well
as the Python3 directories: /usr/lib/python3/dist-packages/ and will
therefore bring in both Python2 dependencies *and* Python3 builds of those
same dependencies.
Installing 2018.2 will therefore bring in python3-django and a bunch of
other Python3 packages, enough to run all the unit tests successfully on
Debian Stretch using Python3, using stretch-backports.
These are not yet used for runtime operations in LAVA, only for development
and testing.
A future release of LAVA will switch to using Python3 at runtime, dropping
support for Python2 at the same time which will also drop support for
Debian Jessie. This will happen *before* Jessie loses security support. As
Stretch will be the minimum requirement for installation, this version
would also move to requiring django 1.11 instead of 1.8 from Jessie.
This creates a window for all LAVA instances to upgrade to Stretch using
Python2 ahead of the separate move to Python3.
To update the summary proposed in June 2017:
Summary
=======
June 17th, 2017 - Debian 9 Stretch release
November 2017 - removal of V1 codebase
January 2018 - migration of Cambridge lab to Debian Stretch instead of Jessie
January 2018 - testing with django 1.11 LTS
2018.1 - testing with python3
2018.2 - add python3 support
Q2 2018 - Release with python3 support, end of support for
jessie-backports and drop python2.7 support. Move to minimum django
version of 1.11
April 2019 - release of django 2.2LTS
Actions
=======
*ALL* LAVA instances should migrate to Debian Stretch support as soon as
practical but certainly in the next three months.
All development related to LAVA must support Python3 as of the release of
2018.1.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
