The master branches of lava-server and lava-dispatcher have now moved to
Python3 exclusively.
This means that all developer builds will only support Python3 and that
Debian Jessie is no longer supported for LAVA software.
It is recommended to use the staging-repo as there are a lot of dependency
changes to add Python3 dependencies on machines which have not used Python3
before. apt will make the job much easier than trying to install developer
packages and then fixing the dependencies manually.
Even if you have 2018.2 installed and have Python2 and Python3 support
available, there are still some new dependencies to move the runtime
support to Python3.
The following NEW packages will be installed:
gunicorn3 python3-dateutil python3-gunicorn
The documentation has been updated:
https://staging.validation.linaro.org/static/docs/v2/installing_on_debian.h…
Once an instance has been upgraded, apt will typically warn about a range
of Python2 packages which are no longer required. These can be removed
using the command:
$ sudo apt-get --purge autoremove
Check the list of packages carefully!
One example list could be:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
python-django-auth-ldap* python-django-restricted-resource*
python-django-tables2* python-ldap* python-lzma* python-markdown*
python-netifaces* python-pyudev* python-simplejson* python-voluptuous*
0 upgraded, 0 newly installed, 10 to remove and 1 not upgraded.
After this operation, 1,855 kB disk space will be freed.
Do you want to continue? [Y/n]
apt is cautious about listing packages for autoremoval, so you may well
find Python2 packages still installed. This is normal.
This starts a new round of functional testing in preparation for the next
LAVA software release. That release will be Python3 only and it will not be
built for Jessie.
Debian uploads from here on will only be to unstable and stretch-backports.
LAVA repositories will only carry updated packages for stretch-backports.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
See
https://lists.linaro.org/pipermail/lava-announce/2018-January/000046.html
The time has come to drop regular builds for jessie-backports. If you rely
on these, you will have found that the builds of the master branch broke
for jessie-backports some time ago (whilst the team were at a conference).
So, in the absence of requests to fix the build and in the light of the
existing plan to drop Jessie support in Q2 2018 (i.e. the next production
release) there seems to be no reason to retain the build.
This also means that building from the master branch for jessie is also
broken - this is not going to be fixed.
This opens the door for testing of lava-server using Python3 exclusively,
dropping Python2 support. Depending on how that testing proceeds, the next
production release could be Python3 only.
Summary
=======
* jessie-backports support in the staging-repo has ended
* developer builds of the master branch no longer need to build on Jessie
(and are currently broken)
* there will be no further uploads of lava-server to jessie-backports or
jessie-backports-sloppy in Debian.
* support for Stretch continues unaffected
* when the next production release is made, a backport to stretch-backports
in Debian is likely to be uploaded.
* There will be no further builds for Jessie for the production-repo or
jessie-backports in Debian.
* testing of full Python3 support can now begin
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Python3
=======
2018.2 is built with Python2 and Python3 support but only Python2 remains
is available for runtime (until Jessie support is removed). This results in
the built packages for Debian unstable and stretch-backports now bringing
in Python3 versions of the existing Python2 packages like python3-django.
The unit tests are routinely run using Python2 and Python3.
The packages available from the LAVA repositories reflect the same:
jessie-backports 2018.2-1+jessie - Python2 only
stretch-backports 2018.2-1+stretch - Python2 and Python3, using Python2 at
runtime.
Bootloader support changes
==========================
25859894 Better detect errors in the bootloader - this adds support to
distinguish between a bootloader failure and a kernel failure to detect
problems when the bootloader tries to start the kernel. This has an
important effect on how some test jobs run - see Quiet Kernels below. The
parallel change (7a2b3a68 Change the flow of bootloader commands so they
are executed individually) supports detecting failures to download
artifacts as distinct from failures to execute once downloaded.
Bootloader action optimisations
===============================
To support the better error detection, several bootloader actions have been
optimised. This means that different actions may be used, changing the
names you may be using for timeouts. e.g.
https://staging.validation.linaro.org/scheduler/job/209814/definition#defli…
The timeout name was u-boot-interrupt - now it is bootloader-interrupt.
The UI shows you which actions have been assembled into the pipeline.
e.g.:
bootloader-interrupt: Wait for prompt Hit any key to stop autoboot (timeout
00:02:00)
Quiet Kernels
=============
The bootloader support changes wait for an indication that the bootloader
has completed and that the kernel has started, by watching for a
kernel_start_message. If your kernel is configured to be quiet, then each
test job using that kernel *must* clear the kernel_start_message:
context:
kernel_start_message: ''
In most cases, the test job should not use 'quiet' as this hides important
debugging information from the kernel boot process.
LXC mocker
==========
lava-dispatcher now builds a second binary package which is intended to
support the use of Docker without needing to change the test job submission
which did not use Docker. e.g. submissions to LKFT use LXC and there are
issues trying to use LXC inside Docker. To make it simpler to reproduce
test jobs when debugging regressions, the lava-lxc-mocker can be installed
in Docker instead of lxc. Commands which would then have been run in the
LXC will instead be executed inside Docker directly.
CAUTION: do not install lava-lxc-mocker *except* inside other
containerisation like Docker. When using lava-lxc-mocker, create a
dedicated Docker image which handles only one DUT so that tools like
fastboot and adb which are constrained inside the LXC do not interfere with
other test jobs and devices.
If you have been using the developer build scripts and you are using 'sudo
debi' then you will need to change to using 'sudo dpkg -i' and listing the
lava-dispatcher package instead of the .changes file.
Release cadence
===============
Now that the roadmap to remove V1 is complete, the need for monthly
releases has diminished. As part of a range of other plans aimed at making
LAVA more flexible, the monthly release cadence has been dropped. We are
looking at a range of options, including wider functional testing of the
master branch, to support releases on a rough 3 month cadence but without
the added pressure of expected release dates. This will become a lot easier
once support for Jessie is dropped and the build matrix becomes simpler.
Changes in this release:
lava-server
===========
9ef41db7 Fix missing failure comment
206e5c65 manage jobs: allow to force fail a job
42a3bc31 Save TestCases even when metadata is too large
5efb92c7 lava-logs: look for lava.job results in all cases
19937074 Use the named urls instead of static urls
e60afcbc Ensure uboot_extra_error_message is usable
843a49ed Use reverse() instead of a static url
7583bb09 device-types: add meson-gxbb-p200
40cbffc8 breadcrumb: keep the last element clickable
2d5f34d8 Update docs examples metadata
5a5548ad device-types: fix broken conditional in base-depthcharge
d663a9dc device-types: add rk3399-gru-kevin
21ac9eb2 LAVA-1230 include lava-logs in encryption docs
d57e9131 device-types: rpi3 32-bit: allow custom kernel cmdline
32312dc7 device-types: rpi3 32-bit: allow extra_kernel_args append
afa28d5a device-types: rpi-3 32-bit: fixup DT load address
0c9e227b Add missing change to the X15 template
968a00dd Fix fdt load command wrapping.
156f1e7b fix quoting of serial numbers
57610c06 lava-logs: simplify log format
5301ef66 breadcrumb: the last element should not be a link
08547de9 Fix api pages titles and breadcrumb
fc03f62b LAVA-1083 - Bug links can be managed only on list pages
35146e4c Remove distinct ON because it can't be used correctly with order
by.
85f6bd67 Test shell helper support with base-fastboot
b86981ad master: do not send START to offline workers
b6cf83f8 admin: fix group cancelation
8890d270 Drop auto-login from hikey 620 template
5642e8cd Allow cancelation in Scheduling and Scheduled too.
bbf6c5bb LAVA-1220 - Update API calls and docs for scheduler changes.
742433b0 Drop invalid boot_message for hikey 6220
441919d7 Skip kernel-start-message for dynamic connections
8f4ecdcb Add a parameter to u-boot to override interrupting
8c093e7f Reorganise fastboot for multiple OS support
abb9fa5d Adding device-type config for MPS devices
240ffeda Add rk3399-puma-haikou device type and health check
bd0c7da7 TestCase: fix crash when accessing old test cases
4e22927e Set Unknown as the default option for changing health
078b6499 api: return the right http error code
e6b3ffdd Fix doc Unexpected indentation error.
b523e20f Fix the filename in header to the actual filename.
acf537fb Improve device tables
a0a87385 LAVA-1225 catch errors when creating health checks
ea8d5a72 Do not show Idle state when device is unavailable
dd4f25bd Allow custom tftp commands for loadfdt in U-Boot
a074403a LAVA-1219 allow unicode for job.original metadata
58d23f23 Python3 branch now merged into master
717c177c fix missing quote mark in docs
dcb8bd5a LAVA-1224 support for Xilinx-ZCU102 ZYNQ
fec11287 Improve python2 and 3 support
756d2e6a migrate-job-output: fix inverted logic for dry-run
fbeaa57f Add NotificationRecipient in the admin site
135654d6 Fix online devices listing
7007fda8 Fix change in 70cb792c7 to show extra_source
025bab80 LAVA-1221 change mouse pointer whenever needed
b28e4c7b LAVA-1222 use verbose names instead of constants
a281a6a7 Extend timeout to cover when dispatcher is busy
3985a0e1 Update development docs for Python3 plans
71374e3a migrate-job-output add --slow and --dry-run
9ce62f2d LAVA-1213 - move developer builds to Python3
81fbb4c4 Charts x-axis attribute should refer to job metadata
7af37d79 LAVA-1164 relicence some lava-server files
14604fb5 LAVA-1211 Document appending to kernel command line
e8eb6ec0 Add admin actions to update device health
b16e197b LAVA-1107 fix "device-types add" documentation
18be0e5b Set device health to unknown after an infra error
54ae8263 LAVA-950 use a ROUTER socket to connect to master
54ed49c2 Fix crash in lava.scheduler.jobs.logs
7a2b3a68 Change the flow of bootloader commands so they are executed
individually
lava-dispatcher
===============
7b9c3e84 Allow some deployments to not use deployment_data
d5dc6418 ssh: fix comparison
ebbd717b Remove spurious conditional in autologin
b3e4d794 LAVA-1232 - LAVA LXC Mocker
4d445841 Fixes for X15 fastboot_uboot
fd6dbec5 actions/deploy/lxc: lxc-apt-install action add non-interactive mode
1621ea5b LAVA-1202 - multiple boot + test section using transfer_overlay
fails
84d78a08 LAVA-1229 - read-feedback: failed on DUTs without telnet or ssh
0ae6815e Ensure RetryAction has protocol support.
6b96c961 LAVA-1227 - Handle cancellation within LXC
ce1421a3 Update ZCU102 device file for bootloader changes
d53e452c Fix login support for second UART
a0104eb9 Remove unused pipeline reference file.
a9f22a9c Allow for empty values in boot methods
00addc73 Fix lookup for grubmenuselector interrupt prompt
782e69d4 Allow skipping of kernel_start_message
52ac66fa Make waiting for a bootloader prompt optional
41f7d58d Not all boot methods have parameters
8b0d03cd Remove deprecated actions in boot sequence.
a3fb0095 Reorganize for hikey 960 AOSP support
7d78da6e Fix usage of FastbootReboot
161c1660 LAVA-1167 - Improve disk usage of test overlay tarball creation
7306a5c6 download: record the http error code when failing
fd1f7013 Pass USB devices to LXC via ID_FS_LABEL
8bae0785 raise an infra error when docker is not installed
414861e1 LAVA-1138 - panic not recorded by auto-login-action
548a6bb1 LAVA-1224 - support for Xilinx ZCU102
abb9869f Unit tests for 960 AOSP support
0a2f84ce Fix man pages
b15eb55e LAVA-950 use a ROUTER socket to connect to master
7a936509 Fix logrotation by using WatchedFileHandler
8f23430b Fix prospector warnings
25859894 Better detect errors in the bootloader
324615d7 LAVA-1165 - skip reading feedback on closed connection
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Roadmap reminder
================
https://lists.linaro.org/pipermail/lava-announce/2017-September/000037.html
Installing 2018.1 will apply the changes from 2017.12 and earlier to
**permanently erase V1 test data** if this has not been done already.
(i.e. if you skipped installing 2017.11 or 2017.12, remember that 2018.1
includes the changes of 2017.11 and 2017.12.)
Removal of V1 test jobs
=======================
A few provisos with this release - there are a lot more changes to the
lava-server package than to the dispatcher because 2018.1 drops the old
lava_scheduler_daemon app and moves all scheduling into lava-master.
The new scheduler state machine involves some database migrations:
Applying
lava_scheduler_app.0032_add_state_health_to_testjob_and_device... OK
Applying lava_scheduler_app.0033_remove_old_status_fields... OK
Applying lava_scheduler_app.0034_worker_last_ping... OK
If your instance has a lot of V1 test jobs still, the migration could
timeout, so as the final part of the removal of V1, follow the guidance in
the documentation to remove V1 test jobs, if you have not already. (By "a
lot", we're thinking more than a few thousand.)
https://validation.linaro.org/static/docs/v2/v1removal.html#removing-v1-fil…
If you have millions of V1 test jobs (as validation.linaro.org had), you
will need to stagger the removal of the V1 test jobs using the --older-than
support. Talk to us if you have issues, there is an updated script which we
used on validation.linaro.org but that would need to be locally
cherry-picked from 2018.1 onto the system running 2017.12.
The removal script can be interrupted with Ctrl-C and restarted if
required. Depending on the performance of the host machine, it may take a
long time to run the removal, up to several hours in some cases.
Finally on the removal of V1 test jobs, the support removes not just the
database object but also the log file directory, so your master should find
a lot more usable fileystem space has appeared, once complete. In many
cases, the log file directory can be missing for a V1 test job but the
warning is just for information. The database object will have been removed.
Removal of unwanted dependencies
================================
$ sudo apt-get --purge autoremove
Do check the list of packages before proceeding. If there are packages
which you need for local support, make sure those packages are marked as
"manually installed" in apt by using apt-get install <package_name> (this
doesn't reinstall an existing package, it simply marks it as not a
candidate for autoremove).
It is worth running autoremove after this release has been deployed, extra
dependencies can still be a security problem and an admin burden.
Stretch
=======
2018.1 *deprecates* Debian Jessie and moves the recommendation on to Debian
Stretch as the supported OS for LAVA.
https://lists.linaro.org/pipermail/lava-announce/2018-January/000046.html
Note that when the instance is upgraded to Stretch, this will bring in a
new major version of PostgresQL and your LAVA database will need to be
*manually* migrated. Neither PostgresQL nor Debian will do this for you.
https://validation.linaro.org/static/docs/v2/debian.html#migrating-postgres…
When following this guide, remember that your versions will differ from the
guide - typically for the upgrade to Stretch, you will be migrating from
9.4 to 9.6. Take care with the commands to ensure you drop and upgrade the
correct cluster. Make sure you have a valid backup. The instance will need
to be in a *maintenance window* for the migration to occur.
Repositories
============
2018.1 has been uploaded to stretch-backports and jessie-backports for the
production-repo on images.validation.linaro.org and jessie-backports will
continue to receive updates until we get closer to the end of security
support for Jessie. See
https://lists.linaro.org/pipermail/lava-announce/2018-January/000046.html
lava-server
===========
d87ccc73 Restore Health and TestJob Reports calculations
6aa325e0 Add a reason when updating device and worker health
aaa1a959 Notifications: use email signature conventions
3f6770f4 Fix notifications and log the exception
fbe57d5f Only use get_change_message on Django > 1.10
832edcf8 fix pep8 error
ff1ef1c4 Disable high limits on ODROID-XU3
7ae61e90 Remove old sample job
4be662da LAVA-1206 - Improve docs for API chunking
6c717e49 LAVA-1207 - Add xmlrpc API call for test suite list.
d8d7ffdd Fix pep8 - E127 continuation line over-indented for visual indent.
81a2a6d1 Return state and health information as strings not numbers.
d2451a1d LAVA-1163 - Unify json library across lava-server.
d9ea9343 LAVA-1192 document use of static_info
824d9580 Deprecate Jessie in favour of Stretch
0791d8be LAVA-1198 - Make sure API covers retrieving the set of test suites
489b83da Add optional support for python3 unit tests
91782fac A sequence is required before excluding a field in table.
6be77f09 AUTHENTICATION_BACKENDS list is available in distro settings only.
938352ff Add documentation for the depthcharge boot method
f6172120 Ensure added blocks are wrapped in conditionals
7ca1a522 LogEntry: improve SQL queries
38c8c8e3 Improve LogEntry tables
e6bb9cf3 LAVA-1199 - Implement incremental result exports in XMLRPC
2945e5f7 Simplify TestJob and Device state display
2e6460bd LAVA-1197 - Document REST API chunking on test suite level.
2f7bc9ab Fix pep8 error
4f896471 LAVA-1197 - Implement incremental result exports in REST API
7a0854ac LAVA-1103 document user and group creation
4b7b353a lava-master: fix string serialization
1e9c1aa4 LAVA-1119 - users and groups for devices
4963ffa9 LAVA-1102 - add group support to manage commands
bf9dd4ec Add missing sub-command parser for device update
9a480bc0 Fix missing import for lava-master
39faeb5f fix crash when canceling a non-scheduled job
4f201dd6 master: use yaml.dump to dump a dictionary
045ed9ec schema: validate multinode
2d2963d9 Allow browsing LogEntry in the admin interface
ba37046c fix pep8 error
8c6b3c35 LAVA-1186 - Fix python3 unit test support
be8b1e5d Worker: record the last ping received
1f4157c1 lava-logs: fix issue when unbinding under python3
d64514fb Port linaro_django_xmlrpc to python3
4b04cc30 lava-server manage: make the sub command required
f49a1bd7 Port scheduler.jobs api to python3
320cb918 jobs: split the database requests in small chunks
babab5ce Prevent noise from apache2 for DocumentRoot
79cfc375 Add FIT parameters to base-depthcharge and rk3288-veyron-jaq
62dd2464 python3: port timing page
b52c84c6 Port BreadCrumb to python3
c4432f55 lava-master: sleep a bit when db connection drop
6b71a7e0 Port lava-master to python3
7f0a2ae5 Improve support for python3
e2182b3c Port lava-logs to python3
cc05ce30 All images that accepts compression will also accept archive.
2bb99530 Port lava daemons common functions to python3
1e84fdc9 Remove deprecated settings
c530a82e remove restrictions on worker health transitions
f66b7f6b workers.show now returns the list of devices
36239079 Print LogEntry messages on the admin main page
02f83e49 Test the Device and TestJob state machine
2d427a84 Remove unused job json functions.
70cb792c LAVA-963 - Support python3 -Wall unit tests
34d52727 Add base-depthcharge and rk3288-veyron-jaq device types
60f85e3e Derive protocol elements from sample jobs.
012034ef Update app_index.html from last django version
849aaa6c manage jobs rm: try to be nice by sleeping a bit
57a2356d Allow for instant canceling
bc754b3f Fix udecode crash when receiving unicode strings
6aad013a Generate an event when a job is submitted
ed9546fc master: improve scheduling regularity
698817ae Fix fastboot boot sequence for hi960-hikey device type.
107cea4b Fix 500: remove_broken_string has been removed
7c95d706 Extend base-fastboot template for fastboot device types.
d779a194 Python 3 support for cStringIO.
65e769d7 Update example test job with pre command
2467411b LAVA-1170 - Unicode literals port
3bf73425 Provide and Document the storage_info support.
270c90a4 Document the state machine and scheduler
87af6a56 LAVA-1169 - Support xmlrpc.client imports in python3
e24865f3 Update lxc fedora example
14792136 No need to save the object returned by get_or_create
6114d0f9 Rewrite from scratch the job scheduler
9da8d83f Fix broken line detection
af632f12 Add a test character delay to rpi3
a5163e61 LAVA-962 - Port lava-server to python3 exclusively - urllib
b7e7ec2a LAVA-1085 - First time 'run query' click does not
enable 'Results' button
a47c170e LAVA-1084 - Clicking on 'live query' tooltip will submit request
e50c2d03 LAVA-1082 - Results action link points to the same page
6705ec87 Refresh the setting files
e4b39c38 Store the lava_server static file in the right dir
lava-dispatcher
===============
b0de4b84 Enabling energy probe data to be passed to LAVA test shell
dd349a77 Use BaseException only when needed
c1a0cd99 lava-slave: when canceling, wait for the job
f00a7315 Port multinode to python3
f937a861 docker deploy: fix python3 port
11e3571d Generate the FIT image for Depthcharge
d46f10c4 Update unit test sample file for images.v.l.o mirror.
b4536e09 Include child devices of matched pyudev devices
eb04f6fd Remove V1-only test shell helper scripts
9f96a0c5 utils/udev.py: lxc_udev_rule only use logging-url arg
when is not None
781312b5 devices/db410c-01.yaml: Add cdt after aboot in flash order
9e809927 LAVA-1172 add support for lava-target-storage
53cfc7db Add support for the depthcharge boot method
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
As announced in June 2017, Python2 is end of life.
https://lists.linaro.org/pipermail/lava-announce/2017-June/000032.html
However, the outline of the timeframe needs a few changes as Debian Jessie
will lose security support around June this year and all LAVA instances
will need to be running Stretch before that happens. It has proved to be
impractical to support LAVA using Python3 on Debian Jessie and the
completion of the migration has meant that lava-server is ready for Python3
ahead of schedule.
Therefore, the pending 2018.1 release will **deprecate Jessie** and move
the recommended installation of LAVA to Debian Stretch.
The 2018.2 release of LAVA will add Python3 support to support developers
and contributors using Python3.
This means that the 2018.2 LAVA packages will contain python code in both
the existing Python2 directories: /usr/lib/python2.7/dist-packages/ as well
as the Python3 directories: /usr/lib/python3/dist-packages/ and will
therefore bring in both Python2 dependencies *and* Python3 builds of those
same dependencies.
Installing 2018.2 will therefore bring in python3-django and a bunch of
other Python3 packages, enough to run all the unit tests successfully on
Debian Stretch using Python3, using stretch-backports.
These are not yet used for runtime operations in LAVA, only for development
and testing.
A future release of LAVA will switch to using Python3 at runtime, dropping
support for Python2 at the same time which will also drop support for
Debian Jessie. This will happen *before* Jessie loses security support. As
Stretch will be the minimum requirement for installation, this version
would also move to requiring django 1.11 instead of 1.8 from Jessie.
This creates a window for all LAVA instances to upgrade to Stretch using
Python2 ahead of the separate move to Python3.
To update the summary proposed in June 2017:
Summary
=======
June 17th, 2017 - Debian 9 Stretch release
November 2017 - removal of V1 codebase
January 2018 - migration of Cambridge lab to Debian Stretch instead of Jessie
January 2018 - testing with django 1.11 LTS
2018.1 - testing with python3
2018.2 - add python3 support
Q2 2018 - Release with python3 support, end of support for
jessie-backports and drop python2.7 support. Move to minimum django
version of 1.11
April 2019 - release of django 2.2LTS
Actions
=======
*ALL* LAVA instances should migrate to Debian Stretch support as soon as
practical but certainly in the next three months.
All development related to LAVA must support Python3 as of the release of
2018.1.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
This release has been uploaded to Debian and the LAVA repositories. The
release fixes a security issue with private test jobs. 2017.12 added
support for the REST API to access test job definitions. This change caused
a problem where test jobs in which visibility had been set to personal or
group were correctly displayed in job tables but the job definitions were
accessible via direct URLs.
This hotfix resolves this problem in lava-server.
There was also a problem with the command line support for adding workers,
this has also been fixed in this hotfix release.
lava-server
===========
2e00544f LAVA-1168 - LAVA job visibility bug
341f5dfa LAVA-1171 - Show 403 instead of 404
31b7b2a9 Fix "lava-server manage workers add"
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Installing 2017.12 and later will PERMANENTLY DELETE ALL V1 TEST DATA!
As per the announced roadmap, the LAVA software team are glad to announce
the release of 2017.12 which marks the completion of the migration to V2
and the death of V1.
https://lists.linaro.org/pipermail/lava-announce/2017-September/000037.htmlhttps://staging.validation.linaro.org/static/docs/v2/v1removal.html#v1-remo…
Upgrades normally try to avoid removal of data but this upgrade
deliberately drops the V1 data tables, permanently. Whilst this procedure
has been tested, there is no guarantee that all instances will manage the
removal of the V1 test data cleanly. It is strongly recommended that all
instances have a usable backup before proceeding.
DO NOT INTERRUPT this upgrade. If anything goes wrong with the upgrade,
STOP IMMEDIATELY and refer to
https://staging.validation.linaro.org/static/docs/v2/v1removal.html#v1-remo…,
then contact us.
At all costs, avoid running commands which activate or execute any further
migration operation, especially lava-server manage migrate and lava-server
manage makemigrations. Remember that removing, purging or reinstalling the
lava-server package without fixing the database will not fix anything as it
will attempt to run more migrations. Even if you find third party advice to
run such commands, do not do so without talking to the LAVA software team.
It remains possible to escalate a failed upgrade into a complete data loss
of V1 and V2 test data by trying to fix a broken system. In the event of a
failed upgrade, the LAVA software team may advise you to restore from
backup and then determine if there are additional steps which can be taken
to allow the upgrade to complete, instead of attempting to fix the breakage
directly. Without a backup, your only option may be to start again with a
completely fresh installation with no previous test jobs, no users and no
configured devices.
Moving to systemd
=================
https://lists.linaro.org/pipermail/lava-announce/2017-November/000041.html
2017.12 also completes the move to reliance on systemd only as PID 1 for
lava-server and lava-dispatcher.
Next stages
===========
Python3
-------
The next large change is the move to Python3. lava-dispatcher continually
passes the unit tests using Python3. With the removal of the V1 codebase,
work can now start on moving lava-server to using Python3 ahead of the move
to Django2 which drops support for Python2.
https://lists.linaro.org/pipermail/lava-announce/2017-June/000032.html
Stretch
-------
In the new year, the Linaro lab is due to move all machines to Debian 9
Stretch. At some point after that, the LAVA software team will drop support
for Debian 8 Jessie.
lava-server
===========
1aeb97ee Default extra_cpu_reset_message cannot be empty.
bd529d6c LAVA-1127 - Extend bootloader reset messages
1c510f12 Fix a data race between lava-logs and lava-master
41c63693 fix pep8 error in pylint comment
aea79d3d Fix authorization for testsuite and testcase views.
3587c07e LAVA-1151 - REST API for individual test case results
23912624 LAVA-1127 - Extend bootloader reset messages
0f523875 Add docs on configuring the services.
971c6bd8 lava-logs: allow to set master_socket and socket
f59e1ab1 LAVA-1123 Add check on active daemons.
da2751b6 Make sure refreshing query works when a migration removes a field.
3fb3272c Handle MultiNode sub_id with the REST API support
06b8797e LAVA-1104 document namespaces
a8e20ece Update device dictionary handling
12597e12 Fixup documentation examples
3682a7f5 device-types: add dra7-evm
c83894fe Fix timeout in query refresh/create because of 'distinct all'
issue.
2498634b fix logging when lava-logs is going offline
549bab84 Send a signal when the worker state/health change
21d68083 Worker: improve main page and make updates atomic
9b6c3365 Worker: make state read only in the admin
fe39f6c3 Fix breadcrumb for devices, jobs and workers
f6ca007d LAVA-1132 document lava-test-raise
d77fec2f Remove unused template tag "linenumbers"
c62734ac LAVA-1125 - Separate results download link for a summary of the
test job
eaeefc82 LAVA-1047 health check issues
c9105768 Fix a typo in code comment.
64bb15c3 Modify apply-overlay and sparse parameter doc in fastboot deploy
action.
b362332f Fix web link to sparse_format.h in doc.
52520b58 Remove unused imports
7c0c1c6c Add a section of required config for UBoot
481c15ec LAVA-986 Poplar support using U-Boot TFTP
731d0419 Add Worker state and health
169bcb0b Fix typo and update URLs in example artifact conversion job in doc.
4a341866 Update various Debian references
8990fd7b Port template to python3
983bc763 Fix Report a bug and Support links.
9b688776 Take CUSTOM_DOCS into account for Help link.
c0cdd0eb Try to create a generator for yaml dumping
974b3a16 Fix help link on index page.
8d344d65 Point to LAVA docs directly without a dropdown for Help.
c7333bf0 Tweak v1 removal page for latest changes
ad8f556d Remove unused functions and dependencies
d6ea4293 Re-organize deploy action doc.
ee2b4dce Remove unused js lib jquery.formset.js.
6e8c08e4 Purge .pyc files on each unit test run.
8c1cc57e LAVA-737 make lava-master the job scheduler
94c1df3d master and logs: always close zmq sockets
e6e5e9da Fix a crash when parsing job description
84172121 Fix a crash when parsing job description
f5b7f16f LAVA-1124 - Adopt StreamingHttpResponse for large downloads
15060b06 Add a YAML syntax checker for static device configs
c98655fd Prevent MultipleObjectsReturned for jobs with many related TestData
2748e682 LAVA-1122 - skip retired devices in deploy check
db2aed27 LAVA-1126 - RESTAPI support for scheduler downloads
f721b3f1 master: improve logging
94a4adf8 Add a simple example for multi-uart test using BBB
c014613c master: schedule jobs only if lava-logs is pinging
7c4570b5 Remove unused file
4aeb14bb Remove daemonise helper
09b1c82c Document the group visibility setting.
e3bbbc5e Remove init.d scripts and use systemd directly
e12b6722 Fix chart query remove link.
09bdae2a Remove obsolete doc regarding test case dependencies.
edddbf9e master: job_status is not sent anymore by slaves
252e2c7a Send the ping interval to the slave.
17ba6914 Rename dispatcher-master to lava-master
2a07e4ea Mark a job as RUNNING only when not already done
7b1cd8c3 lava-master: improve behavior with duplicated END
39e78200 Factorize lava daemon functions
1def9370 jobs: allow to remove only v1 jobs
debec5da Remove unused models and fields
0583c4e2 Add hip07-d05 device type based on d03
3019e41b Move log handling into a separate process
05dbbb7f LAVA-1063 and LAVA-1064 - Drop dashboard app.
42bfd0e5 dispatcher-master: prevent growing message queue
611aaa13 Document the process of V1 test data removal
lava-dispatcher
===============
1f13d442 Fix bug in calculating flash commands order in fastboot deploy.
0b1f8fae LAVA-1155 - Create overlay as part of download deploy action
c20269b9 Prevent crash if logfile is None in shell.
b5244836 ensure lava-test-raise is available for packaging
d3f70511 Fix function signature
0fbaa24f Drop misleading warning message
085a8d49 LAVA-1105 Disconnect from connections cleanly.
32c6f7f4 LAVA-1132 - add an API for abort a test
b5ebe130 LAVA-1127 - Extend bootloader reset messages.
4b140ed4 LAVA-1142 fix VLANd overlay support
bc7fdee8 ApplyOverlayImage init does not take any parameter
d0ff1380 Use lazy logging
334ccaab Never return/break/continue in finally block
9e61e84d LAVA-1130 - Drop unimplemented CustomisationAction
4772dd57 raise an infra error when dockerd is not running
74871923 Indicated when helpers come from the distro
ae5cdce3 slave: "lava-logs" is a reserved hostname
e5713118 Use systemd for slave startup
7ebeb578 slave: do not send status along END message
054fcf9b lava-slave: make ping time configurable
10ab9909 lava-slave: improving logging
6a7bf9d5 Fix wrong duration when timing out at action start
3013b1fa guestfs: change RuntimError into JobError
0fd8a0d0 deploy/fastboot.py: Add sparse parameter to apply-overlay
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
https://lists.linaro.org/pipermail/lava-announce/2017-September/000037.html
Now that the migration to V2 is in the final stages, the LAVA software team
is looking at how the next stages of development will be managed. (The
remaining large items are a clean up of the scheduler and porting
lava-dispatcher and lava-server to python3.)
We'll be retiring bugzilla and moving to using lava-users mailing list for
issue reporting. Members of the LAVA software team will then file tickets
to cover any work involved in fixes discussed on the list. The ticket
itself won't be visible without a Linaro login but there will be an
associated LAVA story which is visible anonymously. The team member
creating the ticket and the linked story will be responsible for updating
the lava-users mailing list with progress and feeding back any comments.
This allows us to collate multiple enquiries into one ticket or create
multiple stories out of one discussion whilst allowing us to properly
prioritise the work.
If you have a Linaro login, you can create tickets in the normal way:
https://projects.linaro.org/servicedesk/customer/portal/1
There is also a public Dashboard:
https://projects.linaro.org/secure/Dashboard.jspa?selectPageId=11603
The dashboard has a dedicated section for tickets and stories which
originated on the lava-users mailing list. The dashboard also gives an
indication of when the next release is due.
The workflow looks like this:
Ticket created Linked LAVA story created.
When work starts, the LAVA story goes into In Progress.
If the work has to wait for other work to be completed, the story and
ticket may be put into Blocked.
When a review is uploaded to review.linaro.org, the LAVA story goes into
Review.
When the review is merged, the LAVA story goes into Resolved with a
FixVersion of the next LAVA production release, e.g. 2017.12 or 2018.1.
Also, the ticket goes into InProgress.
When the release is made, the LAVA story goes to Done and the ticket goes
to Awaiting Acceptance. If no problems arise, the ticket is then Closed.
The LAVA story will be mentioned in the commit message of the associated
review and will therefore show up in the packaged changelog and git history.
There will still be reviews uploaded which are not directly linked to a
LAVA story or ticket but for work which needs to be scheduled, planned or
prioritised, a ticket and story will be created.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
LAVA requires a range of singleton daemon processes to schedule, organise,
publish and log the test job operations. 2017.12 will be retiring the
lava-server daemon and the /var/log/lava-server/lava-scheduler.log, moving
those operations into the lava-master daemon and the
/var/log/lava-server/lava-master.log. This is one of the steps in the
removal of V1.
This leads a substantial change to how admins approach an instance running
2017.12 or later.
When jobs don't appear to be running for some reason, the default action is
to check /var/log/lava-server/lava-scheduler.log - once the instance is
running 2017.12 or later, admins need to remember that it is correct for:
* /var/log/lava-server/lava-scheduler.log to be empty
* the status of the lava-server daemon to be active (exited)
* the lava-server service to not restart
At some point after 2017.12 has been running, admins can choose to archive
or purge /var/log/lava-server/lava-scheduler.log* and / or disable the
lava-server service using systemctl.
The scheduler from 2017.12 onwards will consist of:
* /var/log/lava-server/lava-master.log
* /var/log/lava-server/lava-logs.log
* the status of the lava-master daemon will be active (running)
* the lava-master service will restart when admins request it.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
LAVA, until now, has packaged SysVinit files to manage the various daemons
but there have been longstanding problems with log rotation and the need to
carry redundant code to manage the daemonize process for python scripts.
This has allowed instances to support both SysVinit and systemd because
systemd automatically converts the SysVinit files to systemd service files.
Now that V1 has been removed, it has become possible and advantageous to
drop SysVinit support and move to only systemd support. This is expected to
mean that lava-dispatcher (and therefore lava-server) will depend on
systemd-sysv, i.e. systemd needs to be the init system to be able to start
the daemons upon which LAVA relies.
A default install of Debian Jessie and Stretch already defaults to using
systemd as the init system, so this does not necessarily involve any
changes.
If there are questions or issues with this, please reply to the lava-users
mailing list.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
