See
https://lists.linaro.org/pipermail/lava-announce/2017-September/000037.html
2017.11 is the second release on the roadmap to the removal of V1. This is
the single largest change ever made to the LAVA packages.
* All dashboard URLs are permanently disabled in lava-server.
* All devices which were not enabled for V2 are now hidden and unusable.
* All V1 code has been removed from lava-dispatcher.
* All V1 documentation has been removed from lava-server-doc.
2017.11 disables all access to any V1 test data - including V1 only
devices. The data itself remains in the database but is not accessible via
any API.
If you have chosen to keep an archive of V1 test data and have not yet done
so, you should do this before installing this upgrade. At the very least,
you **must** have a backup of your postgresql database **before** you
upgraded to 2017.11 in order to make a V1 test data archive later.
2017.11 requires that **all** workers and each associated master are
upgraded to 2017.11 for test jobs to continue running. This is because
there have been very large changes to the directories and file layout
within both lava-server and lava-dispatcher associated with the removal of
the V1 components.
The LAVA software team have been checking ahead for the final stage of the
roadmap and plans are on track, as described.
http:// and https:// changes in django
======================================
A reminder, if you are accessing an instance using http:// (whether
localhost or not), then the changes described in the documentation **must**
be applied for the instance to operate correctly. This is a security change
in Django, not a change in LAVA.
https://staging.validation.linaro.org/static/docs/v2/pipeline-debug.html#us…
Symptoms include HTTP500 errors but can also include the inability to login
to an instance using LDAP or local django accounts, depending on your local
apache configuration.
Changes in 2017.11
==================
Changes in the 2017.11 release are described below, including the short git
commit hash. Use the git commit hash to go directly to that commit using a
URL like https://git.linaro.org/lava/lava-server.git/commit/?id=b19b9648a
for lava-server or
https://git.linaro.org/lava/lava-dispatcher.git/commit/?id=1451e3a4 for
lava-dispatcher.
Where the changelog message includes a story id like LAVA-552, that id can
be added to the base URL: https://projects.linaro.org/browse/ to find out
more about the change itself. e.g.
https://projects.linaro.org/browse/LAVA-552
All LAVA stories are publicly available without requiring a login.
2017.11 also includes large changes to the packaging of both lava-server
and lava-dispatcher. The prompts formerly used to configure a V1 remote
worker have been removed. Packaging is maintained at
https://github.com/Linaro/pkg-lava-server and
https://github.com/Linaro/pkg-lava-dispatcher
Packages for 2017.11 have been uploaded to Debian unstable and also the
LAVA repositories (for jessie-backports and stretch-backports).
lava-server
===========
a799fc6c Fix links back to the main instance
5b9656b6 Fix broken plural in header
b21f8c8f Fix crash in job_section_log
c9526f01 Document the LOG_SIZE_LIMIT setting in settings.conf
63b57c94 Remove lava-mount-masterfs
75b403f4 Remove add_device script
8ddea823 Add description about the disc space requirement
for pg_upgradecluster.
5c9dceb2 Remove TemporaryDevice imports
9b21ea48 Removed unused Worker functions
2accb4c3 Matching change to templates for directory change
1d384c4a Update docs regarding device-dictionary lava-server manage
command.
32472fac Port 77fd10e2 to scheduler.devices.get_dictionary
114bbfa5 LAVA-1093 - Add a management command to drop all materialized
views
8bc9d83a Update dashboard XML-RPC API help for api_version
bb090e96 LAVA-552 - add system.api_version support
77fd10e2 LAVA-1092 support passing a context to device_config
de7ad3aa Remove dashboard_app views and static files.
4bc68531 Remove templates from dashboard_app.
1a618c27 Remove unused functions
d50deca7 api: limit the number of jobs returned
59880e14 Fix HTTP 500 on devicetype health history
a39fecf5 Remove unused resources (css, js and images)
a32ac3b6 Rename base-bootstrap and content-bootstrap
7c108de1 job: print lava.job result in case of failure
b51c419f Remove old templates
a9139c1f Use bootstrap template
7915f18f Remove unused resources
7331901a scheduler.api.jobs.show: add failure_comment
71233bff Fix print syntax for python3 compatibility
936d5eca Remove unused tags
4ebb2a39 Remove unused sources
21a7eb18 Save more sql queries by caching values
4d0d867e Remove non pipeline jobs support
a6648dbd Retain docs on disabling V1 workers
8eac0d74 Revert "LAVA-950 set the master identity"
68573b1a Remove V1 dependencies
a8151559 timing: handle new start/end line format
26cc3ad0 Remove unnecessary js lib beautify.
c44009fa Simplify job view
e1da85d4 Fix loading of description.yaml
7b41f7ef Add documentation for secondary media writer parameters
1b4d3dd5 LAVA-1081 migrate instance name to settings
f3dc1987 Fix up typos from earlier changes
faa24f59 LAVA-1079 - remove HIDE_V1 and HIDE_V2 doc options
942b84c3 Remove unused functions and imports
e61a4831 LAVA-1056 Drop V1 documentation
126dce0c Fix crash introduced by 42451c9cc
f0b28214 Remove v1 codebase
31bb1bf0 Fix notification exception when query is used for data
comparison.
a6cff700 Remove last references to lava_dispatcher v1
043690d9 Remove references to lava_dispatcher v1
4c9e493c Remove dashboard_app traces from rest of the codebase.
c371b8ca drop link to removed page
1b1b638e LAVA-876 - Return empty data stubs for dashboard API calls.
919c10c2 Remove links to dashboard_app in the scheduler_app
1483e9fb Remove leftover from job wizard
f7a1c976 docker: allow passing extra options
42451c9c Use all_jobs_with_custom_sort when applicable
cc4ffd64 Fix broken links in example test job
0ca5d75c Drop the migration status page.
b37fe944 Add a device-type config for the rpi-3b in armv7 32bit mode
9e904a18 Revert "LAVA-876 - Remove access to Dashboard"
f07e0950 Fix warning regarding naive datetime
e8348425 Remove more references to dashboard_app
103a44be LAVA-876 - Remove access to Dashboard
d33c9c98 LAVA-876 - Remove access to Dashboard
b6fc46aa Revert "LAVA-1038 add a settings to archive the instance"
f75a333d Do not run dashboard test anymore
d9163445 Remove references to v1 jobs
addb61d9 bug 3268: fix lava-master crash with invalid logs
febf91b1 Improve advice on api/help
ac8b7df3 fix pep8 error
64025cf3 LAVA-1072 test all template connection syntax
10d5fbe1 Adjust U-Boot load addresses for tegra124 devices
to allow bigger kernels
f30ef936 LAVA-1065 - Remove dashboard_app urls server wide.
096a2f1c LAVA-1049 - Allow for .yml as well as .yaml for healthchecks
918f0bc7 Fix logic so addldapuser and mergeldapuser work for
all LDAP configs
36fcb3c2 Documentation changes for multiple uarts
b1de8acf Update instructions for migrating postgres
0ff08373 LAVA-1053 Results limit does not work for queries
f9e64aa1 Fix SQL request storm when listing jobs
2d9c4757 No need to save after get_or_create or Create()
8988cb36 LAVA-896 fix level in result export
219fa3dd LAVA-1073 - support for a second UART on a device
c934bbbe master: fix database reconnection
4f0e0e62 LAVA-950 set the master identity
316d77fb Expand the device integration guidance.
lava-dispatcher
===============
d91a5b99 Revert "Stop parsing kernel messages when the end of a
panic has been found"
1451e3a4 LAVA-1098 unassigned variable in read_feedback
437d57ba Rationalise the devices directory structure
bdffb3bf Add power control to pyocd
d9553680 Add feedback check to test shell
06588ddf Receive the udev device directly from the udev rule
d811b2a8 Fix license
65d6c6a6 Move arduino101 dfu example urls to images.v.l.o
61cff41c Only log when feedback content was found
0ba35392 Move juno removable URLs to images.validation.linaro.org
cb983d97 Cleanup after pytest runs
997265c2 LAVA-1086 add handler to listen to feedback
29a60e38 Add ConfigObj dependency for TFTP check
70bd3fab Fix prospector warnings
6edb8ef9 Fix missing import and other pylint issues
6f7f5f05 Remove lava-lxc-device-* scripts which are unused.
578b054b Use py.test
65c03eca docker: allow passing extra options
7b7d920a device-info should be yaml
ef8f7d4c Remove deprecated function
165e14f8 Fix linger period: set it to 10s instead of 5ms
49c500ac Fix double crash when the process crash early
7dbdc2b6 Add test case for secondary deployment writer tool parameters
969459c6 LAVA-1069 - migrate dispatcher device configs
9e8e5258 Use optargs to extend ci-run for python3
e1c1561b Fix python3 breakage in unit test
69277753 Read feedback when finalising connections
0fb44088 LAVA-1067 Refactor fastboot flash operations
9ac71872 Add missing import
94430958 Ensure finalize always finalises protocols
9d712364 Add option to not uniquify secondary image download paths
cdfd5909 Fix master cert handling for lava-run and lxc helper
73548f7a Rename dd params to tool for removable media
1802c3e8 Handle master and slave certs as filenames only
ed306477 Do not rely on SysV runlevel to ensure container is ready.
75b47c6a Remove loop mounts via guestfs, while applying
overlay to sparse image.
cd1152e3 Support archlinux and slackware rootfs
6fc024a5 Remove deprecated modules
ddf5d2fc Remove v1 code
5561be23 LAVA-1074 - second UART support
758a41a6 Allow configurable writes for secondary storage deployment
fb75db4a Add flag to disable uniquifying the download paths
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Disabling V1 test job submissions
=================================
**IMPORTANT** 2017.10 starts the roadmap to the removal of V1 by DISABLING
all V1 test job submissions.
Queued V1 test jobs will continue to run but once 2017.10 is installed, all
attempts to submit a V1 test job will be rejected. This includes health
checks for devices which do not yet have a V2 health check. (So those
devices would go offline - remove the health check job from the database
entry of the device-type to get the device back online if there are queued
jobs.)
Before installing this upgrade, check the Migration Status page on the
particular instance. e.g.
https://staging.validation.linaro.org/scheduler/migration
Notification on installation
============================
By default, installing 2017.10 will raise a notice, reminding you of this
change:
The 2017.10 release **disables ALL V1 test job submissions**. Queued
test jobs will continue to run but if you proceed, only V2 submissions
will be accepted by this instance.
.
If you want to create an archive of the existing V1 test data, install
this version but do not allow further upgrades of LAVA packages.
.
See
https://lists.linaro.org/pipermail/lava-announce/2017-September/000037.html
If you are deploying this change using configuration management, you may
need to define this environment variable to allow the installation to
proceed:
APT_LISTCHANGES_FRONTEND=none
Window for creating a V1 archive
================================
Once installed, admins can choose to wait until all queued V1 test jobs
have finished and then use a backup of the postgres database to create a
read-only V1 archive. Install 2017.10 on the archive instance and then
restore the backup database. All dashboard objects in 2017.10 are
read-only. Admins will need to ensure that all devices on the archive
instance match those on the original instance and then change all devices
to Retired status in the django admin interface.
See also
https://staging.validation.linaro.org/static/docs/v2/pipeline-server.html#s…
Archive repository support
==========================
2017.10 will be preserved in a dedicated LAVA repository in case an archive
has to be reconstructed from a database backup after 2017.11 is released.
http://images.validation.linaro.org/v1-archive-repo/
lava-server
===========
03fec0ea Fix out of date stretch reference
2c8a88d8 Include non https in title for security defaults configuration doc.
56185add Mark required parameters in left over deploy action reference.
b1de8acf Update instructions for migrating postgres
2b845967 Fix issues in the documentation build.
f9e64aa1 Fix SQL request storm when listing jobs
95b12c01 Fix missing import
0a63c2f3 Adjust U-Boot load addresses for imx6q-sabrelite to allow bigger
kernels
3a4cddf4 LAVA-341 Unicode issues in tables and queries
5bff2fd5 Mark required parameters in deploy action using an asterisk.
00b5044a LAVA-1035 Force all Dashboard objects into read-only mode for users
7a006f64 Change load addresses for Jetson TK1 to allow bigger kci kernels
ca5e6834 Update commands for eMMC boot of OE on X15
0d17e797 Add docker device-type configuration
f0c2107c Remove doc references to lava-lxc-device-* commands.
715c6309 LAVA-771 - Support dragonboard-820c with lxc
7a4adb03 Fix the hierarchy of md5sum and sha256sum in nbdroot index.
576661f4 Make clear when to use lxc://
7c68e21b Document download deploy and explain about lxc:// url scheme.
8f4d7419 Rewrite fastboot deploy action reference doc.
11f5c222 archive: do not allow to force hc
9536450d logger: drop level and action name
771d0d93 Fix a typo in deploy to name.
0a2404fa Update command requires an alias argument
e7a704a1 Add V1 EOL summary to the index page for all docs
cdaa6376 LAVA-1003 do not lose logging messages
f09b6beb LAVA-1045 - document the archive setting
483fb1a7 LAVA-1038 add a settings to archive the instance
866afbf4 migration: v1 health-check are not used anymore
896db15b LAVA-1033 remove v1 job wizard
2eac055b LAVA-1043 fix handling of large log files
aef7d7b8 Add device-type disco-l475-iot1
399b9dbc Use the CDumper to export results
c9dc6a14 LAVA-1032 reject v1 submission at api level
65394986 Drop lookups to ActionData from testcase export
80d1a9fa Add support for the NextThing CHIP Pro
21c8fc52 Only write logs to output.yaml
01df9ce0 Improve job listing by fetching more data
436c220e Decrease SQL queries when listing test results
445b9c66 Add support for reading django settings for limiting LDAP group
access
a33b9c9f Only set the job status if lava-run crashed
581b616f Fix boot action doc for fastboot and lxc boot methods.
08393dbb Use new logging API
33a73bb3 device-types: panda: handle bigger kernels
746fe4c9 Allow to override prompt in device configuration
5932f9d1 LAVA-1048 Extend X15 to typical U-Boot support
affaa35d Fix error message in daemonise.py
e24db160 Fix lambda syntax
8fb16105 v2: Do not fall back to db for health-checks
bceecc64 Do not submit new health checks for v1 only device
lava-dispatcher
===============
1b129396 Remove unnecessary calls to get_udev_devices.
55cc9a0d Replace LxcAddDeviceAction with LxcCreateUdevRuleAction.
9cf899df Include vendor and product id if available, while writing udev
rule.
5eea8b61 Add missing requirement for magic binding
75b47c6a Remove loop mounts via guestfs, while applying overlay to sparse
image.
1f285ecf Ensure the image is Android sparse image before acting on it.
44b8436a Add missing dependency on pytz
b953a562 Append to the udev rules and avoid overwriting.
44380b3e Add a new device type: docker
336719bf Fix broken link in hikey960 unit test.
1703dc94 Remove unused signals after introduction of udev based device
addition.
7f2f049e Always don't assume a ZMQ handler for logging.
9fa3995e Fix a typo in log message.
9cab91d0 LAVA-1046 allow to send logs from lxc udev scripts
37c7e863 log: allow to add a linger when closing socket
dd4f6b09 LAVA-1040 - Initial boot only support for artifact conversion in
LXC
7ed0674c logger: remove level and action name
70fb64c4 lava-run: output_dir should be an absolute path
8f31f584 Always revert directory changes
40ca2c11 Add a timezone aware log message at job start
79325b7b Ensure a retry sets a failed result
b8527df2 Debug log message to know image files are copied to LXC.
3665e840 Allow fastboot calls to fail and return the log
7de0f887 Fix bug #3007 - Unable to reboot target between tests on
hikey-hi6220
dc5e5b18 Send a bad status only when lava-run crashed
18607373 Write description.yaml when all logs where sent to lava-master
e185d967 Move most exception handling to lava-run
57829892 Move signal handling to lava-run
d9d39463 Provide useful output when no classes accept an action
1c45a4ad Setup logging as early as possible
af4401f0 LAVA-1048 extend X15 support for U-Boot
12711359 Raise an error instead of setting self.errors in run()
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
The 2017.9 release is now available from the lava-repositories.
To support Stretch, the package available from the lava-repositories
now has a version suffix matching the suite for which the binaries
were built. So instances using jessie-backports will see
2017.9-1+jessie and instances using stretch-backports will see
2017.9-1+stretch when using the lava repositories.
https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…
lava-server | 2017.9-1+jessie | jessie-backports
lava-dispatcher | 2017.9-1+jessie | jessie-backports
lava-server | 2017.9-1+stretch | stretch-backports
lava-dispatcher | 2017.9-1+stretch | stretch-backports
Additional packages from jessie-backports
=========================================
$ sudo apt -t jessie-backports install python-pexpect
python-setproctitle python-ptyprocess
Make sure these packages are updated from jessie-backports before
installing 2017.9. Versions in stretch are already sufficient.
jessie-backports "sloppy"
=========================
This release has been uploaded to Debian unstable. After a period of
time, an upload will be made into jessie-backports-sloppy which is the
place for backports from buster to the previous stable release. An
upload to stretch-backports will be made at the same time.
To install the new release from jessie-backports-sloppy, a suitable
apt source will need to be added - LAVA still needs some packages from
jessie-backports. e.g. to use the bytemark Debian mirror, add this
line:
deb http://mirror.bytemark.co.uk/debian jessie-backports-sloppy main
Jessie continues to be the preferred release for LAVA but we are
planning to move to Stretch in October. We don't anticipate any
problems with lava-server or lava-dispatcher on Stretch as this has
been tested frequently.
Django changes for localhost
============================
Django upstream have changed some of the security defaults and this
has an effect when using http://localhost - 2017.9 contains
documentation on which settings are required to get localhost working
again.
https://staging.validation.linaro.org/static/docs/v2/pipeline-debug.html#ch…
Reminder about the migration status page
========================================
During the time 2017.9 is installed, all instances need to get to a
similar state of device migration as staging has currently:
https://staging.validation.linaro.org/scheduler/migration
* All active devices migrated to V2 support.
* All active V2 devices exclusive to V2.
* All active devices have migrated to V2 healthchecks
* All active devices have healthchecks enabled.
The important phrase here is 'active'. Health checks can be disabled
for a complete device-type in the django admin interface for that
type, that does not affect completion of the migration status. Retired
devices are not counted at all.
During the changes to remove V1, various of the checks which are done
for the migration status page may also be required to be able to start
or complete the next stage of the removal of V1 - including checks
which could block the installation of a future release if not
completed in time. More details on this will be announced as and when
the checks are created, based on the testing done during development.
The roadmap to remove V1 begins now
===================================
https://lists.linaro.org/pipermail/lava-announce/2017-September/000037.html
**2017.9 is the LAST release which will allow V1 test jobs to be
submitted and executed.**
As of 2017.10, V1 test job submissions will be DISABLED and the
process continues to the removal of access to V1 data in 2017.11 and
the removal of V1 test data in 2017.12.
This means that if an instance is running V1 test jobs after the
2017.10 release, the only support available will be how to migrate to
V2.
lava-dispatcher
===============
255e1dc2 Allow to use lava-run from the command line
5e3e4df8 parser: fix exception with unknown action
b167715b Fix exception message
75ef463a Add transfer_overlay support to fastboot
4fcdbb0c LAVA-1020 - Allow branch to be specified for git
d067a6e4 Remove FIXME about unused feature test-case-deps.
93947b9d Fix job tmp_dir NoneType error.
cf0095ca Extend NBD unit tests for template update
ca72b0cc Honour reboot to bootloader irrespective of power state existence.
b210b297 LAVA-1009 - Improve cleanup of temp files and containers
70fb8c09 Add debug log at the right place for lxc-destroy command execution.
51938bf5 Stop executing shell commands twice.
60c39bae Create device-info-file consumed by udev rules in job's tmp_dir.
46a9789d LAVA-1012 - Handle udev paths outside /dev/bus/usb
c6e06463 LAVA-998 - Default to shallow clones in overlay
b9cc1da8 LAVA-1001 - raise TestError on invalid signal call
4a8241e1 Use check_output instead of Popen from subprocess module.
7024de4c Move to permanent URLs for better unit test support
4f32f4e2 Fix reference to smoke test
927bb33a LAVA-1011 - Check for ENOSPACE after adding overlay to sparse image
47608877 Add udev rule file contents to debug message.
15d8adea Add more debug messages for udev rules addition on the fly.
1b2c9252 Ensure auto-login-action can execute protocols
44755d5a CTT-441 - lava-lxc-device-* don't work properly
d55526c4 Add option to enable IPv6 connections
07839bdf Raise a JobError when lava_test_results_dir is undefined
3f9c87f5 Remove uneeded test
003990f7 Add setproctitle as a python installation requirement to setup.py
d04dcf58 Fix prospector warnings
c2601cbe Extend pipeline_reference readme
115d84ad Create a clean script to replace lava-dispatch for v2 jobs
80bfa16e device: remove 'hostname' and 'target'
b663e427 Do not crash when cleanup raises and exception
fcbff4a6 Hikey 960 support for OE and AOSP
bab3c6f1 Replace kernel.txt with a normal boot and add kernel-panic.txt
c61c67bc Fix value of self.results['success'] in pipeline boot action
6939c5ea Add pre-power command support to LXC protocol
98ede789 LAVA-996 - preserve empty parameters in the overlay
9d33ced5 Use encoding to pexpect.spawn for python3 support
ce2dbd83 Stop parsing kernel messages when the end of a panic has been found
b4ae5b4d lava-slave: Pretty print the configuration
cf22ed83 Extend secondary media to mustang
3e746b47 Add network block device boot support
lava-server
===========
6d1ff5c5 Improve the docs on lava-test-reference and metadata
2f11898f LAVA-997 - Invalidate primary connections setup when power is
configured.
ccfb3961 b2260: allow to set bootloader_prompt in device config
4a1b8f5f Prevent mistaken fixup of device reservation
5093f40c LAVA-1022 - V2 support for IFC6410
1f18eada LAVA-1021 - Document specifying branch for git
36513ec3 Change default source code URL after systems change
da49e558 Remove doc for unimplemented feature test-case-deps.
61dfd204 Document settings to enable authentication in http://localhost
5cbf5b11 Allow vcs checks to skip monitor test actions
b85baa75 NBD kernel bootarg fixups
c089adb7 Add failure comment message via async update on the job detail page.
5ca25620 Improve V1 doc build for Debian reproducibility
dccd1588 LAVA-998 - Default to shallow clones in overlay
250f27dd Fix warning of linaro.img not found.
938b3da1 Fix typos and warnings in deploy to nbd doc.
90c9ed74 LAVA-1013 improve performance of result export
0af80d19 LAVA-1008 Group visibility check
080dd10d LAVA-1007 Prevent NoneType crash in job duration
b2d90962 Fix typos in manage users which talk about devices
07e5fa19 Fix reference to smoke test
59340f70 Expand advice on good commit messages.
cfae2faa Skip some dashboard tests if django is too new
68b24fcc Revert "Move action fields templates to table files."
c9f65507 Add log file argument to dispatcher-master command.
dabaf74c LAVA-1002 - fix group label editing in query details page.
7398abd4 CTT-441 - lava-lxc-device-* don't work properly
f1cb7faf Decrease the number of SQL queries to list the groups
984f3b02 doc: fix example of zmq_client usage
f864696d Documentation build error tweaks
9aeff523 XMLRPC authentication: fix wrong logic
93572067 Basic integration advice for U-Boot devices
5b2d39d7 Support PyOCD for KW41Z and remove CMSIS boot method
3f8c8a93 Fix a typo in persistent container configuration doc.
d334642c Rename xmlrpc module to api
9f947433 Add ssh deployment support for secondary connections.
ce9f0da6 Document how to combine LXC and MultiNode
e2be8567 Allow documentation to be built with python3
cc6e9696 Add notes on writing new unit tests
c40ac0da Document option for device_type qemu to choose host audio handling
97eac4b6 Update lava-self documentation for change in output
7e0f6eca Fix PipelineDevice usage after last dispatcher commit
37d4eba4 Fix a typo - repeated usb_vendor_id parameter in device_info example.
d4a80fc1 Improve the validation message when a role is not found in every
1a5aec5f Add a seealso for fresh installs of lava-tool
4707ff98 Check that the Debian package has been installed
72d30834 Mention cookie specific options for reverse proxy
716424a3 api: fix scheduler.jobs.submit's docstring
6dc1a966 Add sudo for the apache config copy command in V2 worker setup doc.
ae39d57b Add docs on calling protocols from actions
a49cc9c6 Expand the device integration guidance.
1fd5e964 Add Integration Stories for supported devices
620dcbf1 Move smoke-tests into functional test repository
ad2f57df device-type: update at91-sama5d4_xplained addresses
b6734844 device-type: update at91-sama5d2_xplained addresses
c4ead793 device-type: update sama53d addresses
1204f81c Skip links to secondary connections device-types
8b9068a3 Add Metadata output to verbose notifications.
2da8c076 LAVA-357 - Allow override of the nfsvers in base_nfs_root_args
5926b9a8 Catch TemplateError in notifications
98b3da07 Initial improvements for the Queries and Charts docs
092f56a5 device-type: update armada-xp-linksys-mamba addresses
0fdf33a8 device-type: update armada-388-gp addresses
5af8f640 device-type: update armada-370-db addresses
9f4c975f XMLRPC: self.user return an AnonymousUser instead of None
d7b34e2d Add an helper command to switch to a dev setup
9bedff1e Add option to enable listening on IPv6
8ff7c4f1 Extend secondary media docs
f132af4e Add support for secondary media for mustang
1fecd500 Integration of Hikey 960
b27f60c4 LAVA xnbd protocol support and documentation for nbd boot
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
The LAVA software team are happy to announce the roadmap for the removal of
LAVA V1 support, test data and source code.
The next LAVA production release will be 2017.9 which will be the last
release with functioning V1 support. There are no changes to V1 in 2017.9,
so it will continue working as it does currently for this release.
https://projects.linaro.org/browse/LAVA-994 tracks the work for 2017.9
Any instance where V1 test jobs need to continue running must NOT upgrade
beyond 2017.9. There will be no future updates of V1 for any reason and no
support for anything related to V1 other than to migrate V1 test jobs and
devices to V2.
Blocking V1 test job submissions with 2017.10
=============================================
The 2017.10 release will *disable* the submission of all V1 test jobs
either from the UI or using XML-RPC or lava-tool. If any devices still have
V1 health checks, those health checks will no longer run. Any queued V1
test jobs will run. Only V2 test jobs will be allowed to be submitted.
https://projects.linaro.org/browse/LAVA-1026 tracks the work for 2017.10
If you want to keep a read-only archive of your V1 test data, then 2017.10
is the last chance to create this archive. Any instance where the V1 data
should remain visible must NOT upgrade beyond 2017.10.
Removing access to V1 URLs in 2017.11
=====================================
All URLs related to V1 will be removed or disabled in the 2017.11 release -
the Dashboard drop-down menu will disappear and any bookmarks to dashboard
URLs would give a 404. The Dashboard XML-RPC API will become no-ops
returning no data. This means that bundles and bundle streams will not be
able to be downloaded or viewed. Filters, Image Reports and Image Reports
2.0 will all become unavailable.
V1 documentation will be deleted in this release - the V2 docs will be
promoted as the LAVA documentation, removing references to V1.
The V1 source code for lava-dispatcher will be deleted at this point. There
will be some changes to the directory structure of the remaining code, so
if you are maintaining out of tree patches, you will need to fix some merge
conflicts.
Elements of the web UI which distinguish between V1 and V2 support will
also be disabled and removed. This will include removing access to viewing
or downloading the log files of V1 test jobs.
https://projects.linaro.org/browse/LAVA-1028 tracks the work for 2017.11
DELETION of V1 test data in 2017.12
===================================
This has been highlighted many times in LAVA announcements, Connect
sessions, in the documentation and in all support enquiries. It is finally
happening with 2017.12
Any instance which installs 2017.12 will directly cause the deliberate and
permanent deletion of ALL V1 TEST DATA for that instance. Bundles,
BundleStreams, Filters, Image Reports, Image Reports 2.0 - everything in
the database which relates to V1 will be deleted.
Alongside the removal of the test data, the V1 source code will also be
deleted from the lava-server codebase.
*This is a one-way process*. If there is no usable backup for the instance,
there will be NO way to restore any V1 data after this point. After
restoring from backup, if 2017.12 or any later release is installed, the V1
data will again be deleted.
https://projects.linaro.org/browse/LAVA-1029 tracks the work for 2017.12
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
There are delays getting packages into stretch-backports, as expected.
In the meantime, this is a reminder of how to use backports: first
start with stable.
So when installing LAVA on Stretch, even if what you want is the
latest release from production-repo or staging-repo (currently
2017.7), then your first step is to install 2016.12 from Stretch.
# apt -y install lava-server
# apt -y install vim
# wget http://images.validation.linaro.org/production-repo/production-repo.key.asc
# apt-key add production-repo.key.asc
# vim /etc/apt/sources.list.d/lava.list
# # edit the file to specify: deb
http://images.validation.linaro.org/production-repo jessie-backports
main
Yes, that is jessie-backports - we don't have packages in
stretch-backports at this time.
# apt update
# apt upgrade
If you specify backports too early, you'll get a range of other
packages from backports - you don't actually need jessie-backports or
stretch-backports from Debian at this time.
Packages for jessie-backports are built on jessie. Packages for
stretch-backports are built on stretch. It's the same source code in
each case. Right now, there aren't any problems with installing from
jessie-backports on stretch - particularly if you install lava-server
from stretch in the first place so that the majority of your packages
come from stretch.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
During testing, we've seen that upgrades to Stretch from Jessie can
have an issue if UWSGI is still left from a previous installation:
apache2: Syntax error on line 146 of /etc/apache2/apache2.conf: Could
not open configuration file /etc/apache2/mods-enabled/uwsgi.load: No
such file or directory
Action 'restart' failed.
The Apache error log may have more information.
dpkg: error processing package lava-server (--install):
subprocess installed post-installation script returned error exit status 1
Setting up lava-server-doc (2017.7-1~bpo9+1) ...
Setting up lava-dev (2017.7-1~bpo9+1) ...
Processing triggers for systemd (232-25) ...
Processing triggers for man-db (2.7.6.1-2) ...
Errors were encountered while processing:
lava-server
The problem is because of a bug in the uwsgi packages. The uwsgi
module is not cleanly removed from apache. Run this command to fix:
$ sudo a2dismod uwsgi
Then complete the install:
$ sudo apt-get -f install
Fresh Stretch installs should not be affected as 2016.12 in Stretch
does not use uwsgi.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
The 2017.7 release is available from the lava-repositories for
jessie-backports and has been uploaded to Debian unstable. Support for
stretch-backports is pending. (The production-repo/jessie-backports
can be used in the meantime.)
https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…
jessie-backports "sloppy"
==================
This release has been uploaded to Debian unstable. The period after a
Debian stable release is turbulent in unstable and the LAVA packages
may not migrate into buster (testing) smoothly which will delay
uploads to the official backports archive (for jessie backports and
stretch backports).
To update jessie-backports, the upload will need to be the version
from buster, rebuilt for jessie and sent to jessie-backports-sloppy.
Once that is done, I'll post again to the this list about how that
becomes available to admins running Jessie systems.
Jessie continues to be the preferred release for LAVA until such time
as we've done more testing on Stretch. We don't anticipate any
problems with lava-server or lava-dispatcher on Stretch as this has
been tested frequently. However, third party packages and scripts
still need testing with real instances using Stretch.
V1 support in buster
===============
Some unit tests in the V1 support have had to be disabled due to bug
fixes in django 1.11 LTS in buster and sid affecting the atomic
transaction handling. The remote worker design in V1 has always had
issues with atomicity and these unit tests have failed intermittently
for a while. This is one of the many reasons to drive all instances to
V2. These tests are now failing consistently with 1.11, so have been
disabled if django 1.11 or newer is detected.
V1 is no longer run on any of our development instances (running
buster or sid), only in production instances (running jessie) which
are in preparation for the migration. We can only assume that the
failure of the unit tests would make V1 unreliable on buster or sid.
However, with the pending removal of V1, there will not be any fixes
for any problems discovered when using V1 with django 1.11 or newer.
LDAP support in buster and sid
=======================
python-django-auth-ldap has changed dependencies, from python-ldap to
python-pyldap. When upgrading python-django-auth-ldap, python-ldap
will be removed and python-pyldap installed.
Instances *must* restart the lava-server-gunicorn service after this
change or users will experience HTTP 500 errors.
lava-server manage check --deploy
=========================
This release adds some checks to the django system checks. When admins
update health checks or device-type templates, it is worth running the
deployment check which does a quick sanity check on the instance. This
can help to spot invalid health check submissions amongst other
issues.
Note that on development instances, some issues will always be
reported - e.g. when Debug is set to True for django using
/etc/lava-server/settings.conf
Also, for developers, some of the improved security features in django
may get in the way of signing in to http://localhost by enforcing
requirements for https:// which is unusual on localhost.
Check the django documentation - to enable localhost, you may need to
disable at least these security defaults by adding to
/etc/lava-server/settings.conf:
"CSRF_COOKIE_SECURE": false,
"SESSION_COOKIE_SECURE": false,
As a result of these changes, installing 2017.7 can generate a prompt
to update /etc/lava-server/settings.conf. In general, keep your
existing configuration but look at the differences and migrate the
changes across. Remember to restart lava-server, lava-master and
lava-server-gunicorn services whenever /etc/lava-server/settings.conf
is modified.
Skipping 2017.8
============
There won't be a full 2017.8 production release of lava-server or
lava-dispatcher as a number of the team are away at the annual Debian
conference, DebConf17 in Montreal.
https://debconf17.debconf.org/
If you are going to be there, let us know.
lava-server changes
==============
a9390fc9a Remove debug hacking session from doc examples
13edd8f12 Skip certain V1 unit tests on django 1.11 and later
0577447fc Update query script for python3 and to output CSV
0b3b0beb7 device-type: addresses fix for armada-3720-espressobin
ed1e402e5 device-type: addresses fix for armada-3720-db
c382a5aa0 device-type: fix at91sam9m10g45ek
54a3b7a46 Apply change for distinct test case links to all results
1824a18fb Adding uefi-menu tftp method
71aac817e LAVA-363 - set the scheme in api/help from settings
2c840debc base-uboot.jinja2: don't load DTB if append_dtb is set
1b13a0e3e Fix extra keyword in api/help view
2882529bc Provide an example script for using queries
7f4cfe569 logs: continue to poll the log when canceling
25eaab176 XMLRPC: add offline_info argument to list endpoint
38f7570e3 XMLRPC: fix inconsistency between code and doc
8c134839d LAVA-972 test case should use ID in URL.
99725606c Remove references to OpenID and Crowd
95b959602 XMLRPC: add a function to download job definition
ebf80c266 XMLRPC: fix crash when device.worker_host is None
b1ec14c89 LAVA-861 separate base-fastboot.jinja2 template
c4fd79d57 Remove unused file
7c066195a Remove deprecated django configuration
520ac8778 XMLRPC: remove unused function
3b8ab7d61 XMLRPC: do not raise an error when canceling twice
cce1bdea6 XMLRPC: allow to force health check for a device
218408db6 Handle the new log level called "input"
b49d9fb96 Adding uefi parameters for vexpress
4a52ba861 Allow overriding django system check values from settings as
documented
d2067a5f3 LAVA-987 fix log parsing
bbc39489c results_app: API: add the 'run_query' function
8746edb6b results_app: API: add the 'make_custom_query' function
4fd429ac2 Add note on apache2 for V2-only workers
641e61dcb Explain the handling of measurements and Decimal
bf8f53bf5 Count going offline as busy in overview table
06febd188 Expand on essential roles, results and sync
815d41b52 Tighten advice on prompt strings in docs
351f6fe41 Add documentation for login_commands in auto_login
5a7d4be95 Avoid warning for health checks of retired devices.
5713f5295 Document transfer_overlay deployment differences
8f04c6097 Fix typo in multinode api documentation.
fe2093994 device-types: add meson-gxl-s905x-khadas-vim
853ee80e3 Move action fields templates to table files.
12de946eb Allow bare except and override pep8
dd4faacaf Enforce schema rules for metadata
352049204 Add missing check for null key|value in metadata
b77731b31 device-types: rcar-gen3-common: handle bigger kernels
d2acb94a5 LAVA-983 - Helper for DeviceStateTransition
b4208116c LAVA-980 - Fix django deployment warnings
778fd4a8b Fix link to glossary term
af85e5225 LAVA 931 - documentation.
212e98a3a LAVA-976 Use the Django system check framework
e9c7ad477 Adding commands for booting whatever the precanned defaults are
f78c58811 Added device type for Cavium ThunderX
lava-dispatcher changes
=================
d8dabe63 Switch the print to a test in unit test
d1e22b67 Revert "Replace invalid characters in test_case_id."
b3b4f5ac Update the example to a working test job with prompts.
504ec2bc LAVA-777 last log line should be lava.job result
ad55acb9 Ensure overlay unpacking is clearly logged
2e7c9598 Add pipeline_ref update for test skipped by lavabot
ed936cd3 Simplify the power control actions
78dcc1b0 Fix dispatcher configuration
89880ce1 LAVA-96 Check for errors if a tftp file exceeds 4G
6fd2702e LAVA-989 add extra logging for actions
df86ed30 LAVA-990 add logging for VLANd + MultiNode support
bbfb1817 Stop the unit test outputting noise
c7839a98 Add a new log level for the DUT inputs
2502b8af Destroy LXC containers properly.
957734cf Add a unit test for Vland actions
68468abe Handle a bad url cleanly.
713d4e6e LAVA-645 - summarise slave logging
96a5effe Do not raise exceptions in validate
f45106fb Catch errors in testdef from handler cleanly
f34f41e2 LAVA-984 - LXC protocol timeout to report job result
e31722b5 Allow ci-run with updated pep8
175d8e84 LAVA-931 - MultiNode lava-sync should report results
fd944543 EOF in a test shell is an InfrastructureError
429691d9 Support power control for CMSIS devices
049c0227 lava-slave: add the right configuration for the host
1e92bf38 lxc: improve error message
b2f25cf0 Add support for booting from UEFI shell
67a1e1c6 ramdisk: only pass to bootx if has u-boot header
f4a4abd9 LAVA-967: check namespaces usage
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
As with previous releases since 2016.12, the 2017.6 release is only
available from the lava repositories:
https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…
However, Debian Stretch is due to be released very soon. 2017.6 may be
the first release to be uploaded to Debian unstable since the stretch
freeze started with 2016.12. There may be a short delay whilst
unstable settles down after the usual round of extra uploads. Once
packages migrate into the new testing (buster) and stretch-backports
becomes available, packages will be uploaded for stretch-backports and
jessie-backports.
Device dictionaries moved to the filesystem
===========================================
LAVA-757 - all existing device dictionaries will be migrated to the
filesystem (/etc/lava-server/dispatcher-config/devices/) during the
installation of the 2017.6 production release. The dependency on
python-django-kvstore has been dropped. This means that there is no
longer a need to update the device dictionary in the database, simply
install the new file into the directory and the next test job to be
assigned to that device will use the new configuration. Unlike
device-types, there are no default device dictionaries provided by
LAVA, so a fresh install will have no device dictionary files in
/etc/lava-server/dispatcher-config/devices/
The documentation has been updated. There is now no need for
lava-server manage device-dictionary, so this command has been
dropped. Instead, admins who have permissions to run lava-server
manage can simply copy a device-dictionary .jinja2 file into
/etc/lava-server/dispatcher-config/devices
During installation, this migration will take place:
Applying lava_scheduler_app.0027_device_dict_onto_filesystem
Database entries for all device dictionaries are replaced with files.
If a device can support V2 but has no device dictionary, it will be
marked as Skipped.
This change means that lava-server no longer depends on the
python-django-kvstore package.
Feedback support with namespaces
================================
Devices may output messages to the serial console as a result of
operations performed within an LXC and this output is now logged as
part of the test job log file as 'feedback'. See the documentation for
more information on feedback output.
Architecture is optional for LXC
================================
Some LXC deployments support identifying the deployment architecture
based on the host system. Some instances may use workers of different
architectures and test jobs should be portable between instances.
Therefore, the arch: field of the lava-lxc protocol has been made
optional. Note that Fedora and Ubuntu deployments still need the
architecture to be specified for x86 machines to distinguish between
32bit and 64bit x86 deployments.
The architecture value can still be specified in test jobs but the
test jobs will be more portable if omitted.
Action names use hyphens, not underscore
========================================
If you are using action-specific timeouts in your test jobs or in
customised device-type templates, note that the names of some actions
have *changed*. This is to make the usage of hyphens and underscores
consistent. Actions with names containing underscores are now invalid
and all underscores in names have been replaced with hyphens.
'pdu_reboot' becomes 'pdu-reboot'.
lava-server changes
===================
daf1e2a lava-master: fix init script
b6b5fe7 Reduce device configuration dynamic connections
ebf4463 Fix a crash in the migration when the db is empty
cdfeb12 Fix index out of range exception
33092ef Adding usb deployment method to vexpress device template
3257ee2 XML-RPC: remove unused imports
55dd014 XML-RPC: fix crash when called by anonymous users
a3d4718 XML-RPC: allow every user to get templates/dict
ddaf693 Add documentation of feedback support.
eaae89a Set a default connection timeout for lava-test-shell
f305e47 publisher: fix umask
c25686f logrotate: lava-master should be owned by lavaserver
2da8e74 logs: except more exceptions
14f5f6c Update docs for changes in 2017.6
a486d12 Tidy up of debian support docs
b05712d Preserve original comments when resubmitting
19bd2db XML-RPC: add an api to manage aliases
30973a0 Log an error when saving device configuration fails
763239d Use --arch for fedora LXC example
8f26102 XML-RPC: add a method to show tag details
19a69f7 fix location of example test job files
480a6f6 XML-RPC: add an api for handling jobs
356667d Update installation docs for Stretch release
da74ae5 XML-RPC: add a decorator for is_superuser check
49f370c XML-RPC: add a clean api for device-types and aliases
c6d503d commands/devices: extend devices command to add tags
1655dbf Move set commands outside body block
87b8c01 XML-RPC: add a clean api for workers and tags
351c16b XML-RPC: add a clean api for devices
c4dec1f device-types: add bcm2837-rpi-3-b
9ca0e0d Adding combined vexpress device template
67f531f Make lava_scheduler_app.api a module
d3021a3 xmlrpc: add a system.version function
3ef0157 linaro_django_xmlrpc: remove unused properties
e994852 Use render shortcut instead of the loader
e51ca21 linaro_django_xmlrpc: rework the application
4bbe17c LAVA-959 run lava-master as lavaserver user
b283288 Fix migrations from Debian Jessie to Stretch and ensure smooth upgrade.
bc25204 LAVA 329 - Add old job URL to metadata when resubmitting a job.
271723d LAVA-782 - Change error type for incorrect handling of custom queries.
368903e notify: Don't send verbose mails when the job is cancelled
71c514d Fix broken link in unit test.
1f48674 LAVA-955 - Remove 'arch' parameter from lxc protocol.
3b36ba0 Add auto_login test case in test_pipeline unit test
e23bb17 Fix typo which causes XML-RPC error when saving device dict
9a82aee XML-RPC: allow to have sub_modules
b0c7a5b Document reboot and apply-overlay elements for fastboot deploy method.
8a93049 LAVA-935 - Provide feedback output from connections
d71f267 Fix typos in proxy configuration documentation.
c7227b3 Fix rebase error in grub hikey test
9253497 Only the master should create log directories
1ee736b Allow every u-boot devices to use minimal boot
e816c17 LAVA-832 allow to manage users from the command line
eb83cbe Add grub interrupt prompt and interrupt character to constants
ee79de2 LAVA-937 show job visibility and fix group visibility
73b6d86 Prevent key error in metadata handling.
f89e840 Improve job output migration helper
eb1ad02 device dict: do not show exclusive flag
3900276 Restore date-based subdirectories
1faa9f0 LAVA-855 - ART CI: Intel NUC device integration for LAVA v2
d195b01 Allow to revert (dummy) the migration
35f0802 Fix device dict export
da7ebe2 device dict: fix web rendering
4a3daa9 Allow override of root for installed mustang.
12c0508 Expand zmq_client to handle publisher socket
34f3625 Trivial whitespace changes in glossary
6246a5d Improve data export docs
2af8e72 Allow for non-Const values in device dictionary
59742fe Extend examples to use the DNS support for NFS
c514296 Fix change in import paths for V1 and V2 scheduling
765901e Add notes on advanced features for submission
fd0905c Remove deprecated commands
0237b85 Use CommandError whenever possible
a1cc08b Improve device-type templates
05123cc Missing conditional in device-type alias handling
17af14e Move commands to lava_server app
5bab54d Remove reference to unavailable boot method 'fastboot-boot-image'.
78e9048 Add mediatek dt for v2
49f27fa Allow in-place YAML include in V2 job submission.
c2be83f Grub support for HiKey
ae4d477 LAVA-757 Move device dictionaries to file system
52eb1e1 Remove unused model JobPipeline
9d0a5ae results: make the link unique
55bf39f Add a callback_url functionality to notification schema.
0324915 Add login_commands to the auto_login schema
ce6d0f7 Don't reload template from disk on each access
lava-dispatcher changes
=======================
38da5d8 LAVA-974: Fix init script and allow to set hostname
0fd687e Don't use os.path.realpath() on device names
d84dab7 Modifying vemsd deploy action to use ResetDevice
d21385e testdef: get 'revision' from the right variable
5ea16fe Rework test shell feedback timeouts
510c4e5 Declare the feedback timeout
b994757 Use pyudev device links as well as device nodes
e609c22 Fix namespace usage in power commands.
39e2e7e Read from feedback connection during long operations
7be926a Allow tar.gz firmware bundles
20e66c1 overlay: Raise JobError if root_partition is None
4e27fcd LAVA-935 - Provide feedback output from connections
831c71c Fix bug #3032 - Running an LXC/Hikey job without a test action ...
84e7378 Fix namespace issue in NFS deployment action
f1f8d71 Add login_prompt valid test case
7dc48aa Issue lxc-stop before destroying a container.
3e1cb07 Changing udev checks to only look for FS label
d08db39 Fix #3010: set logging timezone to UTC
1b664e5 LAVA-955 - Remove 'arch' parameter from lxc protocol.
8a4dbc7 Increase LXC finalize timeout to 90 seconds similar to multinode.
3823c49 Use one debug log call to log the same kind of message in lxc protocol.
5db313c Creating vemsd mount points
ea03ef1 Fixing boot test issues using NFS deploy action
d1f4a30 Fix rebase error with grub prompt constant
b2aef8b device-types: add sun6i-a31-app4-evb1
8ceca07 Make grub use interrupt prompt and character from constants
b495a35 device-types: add ttyAT0 to Atmel devices to support multiv7_defconfig
3a89027 device-types: fix at91sam9x25ek to use bootz
9e00895 Set download max_retries in the constructor
959480c Add minimal boot action
31fd021 User command: do not clean if it was not executed
26572a2 Fix keys and label that were outdated
67dd60e Remove unused functions
0c100ff Remove boot-result common data
342039a Fix RetryAction inside a RetryAction
9ce430f Fix check on LXC for default usage.
0576754 Fix typo: Use pixel job create function for testing pixel device.
e7a4422 Fix typo: remove visibility added twice.
9e67b6d LAVA-855 - ART CI: Intel NUC device integration for LAVA v2
dd5d874 Ensure all deployments can add LXC devices except QEMU
e72d490 Fix pipeline refs for change in monitor retry
cb57d2b LAVA-942 UEFI Menu and command lists
e57013b Start adb daemon before attempting adb commands.
ac389ad lava-slave: remove tmp dir when END_OK is received
632a0ff Fix device configuration
0cffce4 Fix validation of boot auto_login parameters and add a unit test
7e40701 Allow unix line separators for some UEFI devices
095798a Avoid calling cleanup in validate
f98dade Allow updating all pipeline_refs at a time.
31b62dd LAVA-928 - HiKey issue when switching between AOSP and OE
51f40ef Make soft reboot optional LAVA-846
4d43d02 HiKey Grub EFI support
874f889 Avoid logging the same message twice.
3be27df lxc protocol: simplify the tests
75126ff Do not set self.errors when Action.run fail
ce32bfb Do not use self.errors in Action.run()
a5d9eba Fix detection of missing ssh_host value in validate
234eef9 Fix calling of protocols after LXC change
477a36b tests: make ShellCommand.logger a DummyLogger
62867f4 Use images.validation.linaro.org files for unit tests
1868936 Silence logging in more unit tests
609c783 Fix download action name
b170197 More tweaks to silence messages from the unit tests
29efa26 Adjust for pep8 checks in jessie
0733f89 Make sure the test_character_delay is used for all commands in
test shell
78d2509 Account for empty environment string.
ea09b19 Take namespace into account when counting test stages
734688c Declare namespace of the test suite in results
60561c2 Parser: remove unused context
49a6b92 Drop noisy info log message.
8ed8467 lava-test-monitor: use TestMonitorRetry action
4fd5d4a Drop unused imports and unused variables
678b216 Rework the removable action to allow sd cards
3b42122 Fix bug #2975 after qemu-nfs introduction
7462ea6 Expand LXC support to add devices from all sybsystems
b853865 Fix missing check for u-boot commands parameters
8293693 Replace invalid characters in test_case_id.
a94e0f2 Fix preseed/late_command appending
9647d55 Run login_commands if provided in auto_login
db03483 Fix Action names (use - instead of _)
9e836e6 Add a Command Action
bbd4ba2 Make the parser stricter about the block names
65b3a56 Allow to pass integers to run_command
b41326b Raise JobError instead of NotImplementedError
073f3b3 update gitignore
00f5d33 log: limit the length of lines send other zmq
3f99cc3 Add missing super call to WaitUSBDeviceAction
5b2b074 LAVA-889 Fix handling of multiple test blocks
5436658 Fix indent typo
3bfdcdb Add support for use_xip
8df17dd Add support for append_dtb
3c110ca Remove unused imports
4847bf1 Put device rebooting info message at the correct place.
6b85382 Fix call to lava-lxc protocol for pre-os-command
176a4d4 Add new utility function infrastructure_error_multi_paths
bdb4d12 Fix test_lxc_api unit test
dc008eb Export the full version string.
e9df320 LAVA-920 - Workaround ptable issue in firmware
7b4096c Allow LXC protocol to call pre-os-command
b33d04b Fix OE image support for HiKey.
d714a9b Add missing test shell helpers
daebfad Extend secondary connection fix to support primary
9518e74 Add missing import
e97801d Add optional deployment complete message list to secondary
media deployments
fae0ced Move unnecessary constants into base jinja template.
18f745b device-types: add kirkwood-db-88f6282.conf
815e518 device-types: add at91sam9m10g45ek.conf
6ca0731 device-types: add at91rm9200ek.conf
003b53d device-types: add at91-sama5d4_xplained.conf
9b2371c device-types: add armada-3720-db.conf
5a76733 device-types: add armada-xp-gp.conf
8b05093 device-types: add kirkwood-openblocks_a7.conf
f5801d3 device-types: add alpine-v2-evp.conf
19e2027 device-types: add sama5d34ek.conf
49da704 device-types: add armada-385-db-ap.conf
ed7aa6b device-types: add armada-370-db.conf
9a10432 device-types: add at91-sama5d2_xplained.conf
ddb34ce device-types: add alpine-db.conf
f7642f5 device-types: add armada-375-db.conf
0d70c7d device-types: add at91sam9x35ek.conf
a642d45 device-types: add armada-xp-db.conf
2a5ff9e device-types: add armada-7040-db.conf
1a43999 device-types: add orion5x-rd88f5182-nas.conf
ad855a6 device-types: add armada-xp-linksys-mamba.conf
58308ca device-types: add armada-388-gp.conf
6a2898b device-types: add armada-370-rd.conf
cd1b4f4 device-types: add armada-398-db.conf
51925fb device-types: add sun8i-a83t-allwinner-h8homlet-v2.conf
4976d06 device-types: add sun8i-a33-sinlinx-sina33.conf
681480d device-types: add sama5d35ek.conf
db74d60 device-types: add sun5i-r8-chip.conf
40736b9 device-types: add sama5d36ek.conf
433be0a device-types: add imx6q-nitrogen6x.conf
ed4c82e device-types: add at91sam9x25ek.conf
ce7a488 device-types: add at91sam9261ek.conf
f87cdb4 device-types: add armada-xp-openblocks-ax3-4.conf
634442c device-types: add armada-388-clearfog.conf
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
This is a long mail but there is a lot to cover...
Stretch
======
Debian Stretch is expected to be released on June 17 2017:
https://lists.debian.org/debian-devel-announce/2017/05/msg00002.html
As has been mentioned on this list before - admins MUST check the apt
sources for all machines *NOW* to ensure that all your Debian apt
sources are pointing at 'jessie' NOT 'stable' if you are to be able to
manage the upgrade yourself.
What you want in /etc/apt/sources.list and /etc/apt/sources.list.d/* is:
deb http://mirror.bytemark.co.uk/debian jessie main
deb http://mirror.bytemark.co.uk/debian jessie-backports main
(The mirror you choose is up to you.)
If you find any apt sources referencing 'stable' or 'testing' CHANGE
THESE NOW or those systems could automatically upgrade to Stretch or
Buster respectively on 17th June.
As with all such releases, production machines are recommended to
allow some time for final testing before doing the upgrade. The
Cambridge LAVA lab is expected to upgrade from Jessie to Stretch
during August 2017. The LAVA software team have been doing testing of
this upgrade for some time, without issues. However, real machines
with extra packages and local admin configuration cannot be tested in
this way, so each lab should take some time to plan when and how to do
the upgrade. (Hint: have a backup and check that it can be restored
correctly before starting.)
https://validation.linaro.org/static/docs/v2/admin-backups.html
There has been a fix for lava-server in stretch to smooth the upgrade
for those users who have not installed lava-server from
jessie-backports. Those already running 2017.* are not affected by
this change and can continue using jessie-backports and the LAVA
repositories as before. Stretch will release with lava-server
2016.12-2 and python-django 1:1.10.7-2
Once Stretch is released, Debian testing will be called 'buster',
Debian stable will be 'stretch' and Debian oldstable will be 'jessie'.
At a later date, jessie-backports will be opened again and a new
stretch-backports will become available. New LAVA production releases
will be uploaded to unstable to migrate into testing (buster) and then
be backported to stretch-backports and jessie-backports. The LAVA
repositories will be updated as well.
Support for jessie-backports will continue after stretch-backports
support is added but cannot continue indefinitely. At some point, all
instances will need to have moved to Stretch to continue getting
updates of LAVA. (Prior to the process of releasing Debian Buster, the
jessie packages will be dropped from the archive, including
jessie-backports.)
Changes in LAVA repositories
=======================
2017.6 will be released in the same way as previous releases in 2017 -
to the LAVA repository for use on jessie using jessie-backports.
At a later point, 2017.6 will be uploaded to Debian unstable for
migration into testing (buster). Only developers should install that
version as Debian unstable can be problematic in the first few weeks
after a Debian stable release.
There are changes in how the lava packages are built on jessie versus
on stretch, so the lava repositories will be split. I'll be updating
the documentation in 2017.6 to cover the changes required to help
admins use the new suites. There will be a new jessie-backports suite
and a new stretch-backports suite. The existing 'sid' or unstable
suite will have no uploads after 2017.6.
This will be covered again in the 2017.6 release email and again with
2017.7. For now, no changes are required.
Once the new suites are announced, admins who choose to upgrade to
Stretch would use the stretch-backports suite from the LAVA
repository.
deb https://images.validation.linaro.org/production-repo stretch-backports main
Those who will upgrade later, would use jessie-backports from the LAVA
repository.
deb https://images.validation.linaro.org/production-repo jessie-backports main
The purpose of these suites is to allow admins to choose to upgrade to
the latest production release for jessie or stretch without waiting
for the migration of the packages into Debian testing and the
subsequent uploads to backports. The packages are otherwise the same.
Django
======
Longer term, there have been a lot of discussions within Debian about
managing django releases. The plan is that only django LTS releases
will be available in Debian unstable, testing and therefore stable
after the Stretch release. Non-LTS releases will only be available in
Debian experimental. This means that lava-server will migrate from one
django LTS to the next django LTS with future Debian releases. The
LAVA software team will manage how deprecation cycles within django
affect lava-server.
This does have the issue that the version of django in Debian stable
will be towards the end of the upstream support cycle but it is better
than having a non LTS django release in Debian stable. Synchronising
releases across multiple upstream teams is not feasible.
The next django LTS release after 1.8 is 1.11 which is expected to be
in the next Debian stable release - buster.
https://www.djangoproject.com/download/
Python3
========
The last django LTS release to support python2.7 will be 1.11 LTS
which retains upstream security and data loss bug fix support until at
least April 2020.
The LAVA software team have already been testing lava-dispatcher V2
with python3. Once the V1 code is removed from lava-server, testing
will proceed for lava-server too. We fully expect to be able to
support lava-dispatcher V2 and lava-server V2 on python2.7 and python3
for a limited time.
There will never be python3 support in lava-dispatcher V1 or lava-server V1.
Python2.7 support for lava-dispatcher V2 and lava-server V2 will be
removed at some point prior to the release of the next django LTS
after 1.11 which is 2.2 LTS, due in April 2019.
Once django 2.0 (non-LTS) arrives in Debian experimental, the LAVA
software team will begin testing lava-dispatcher V2 and lava-server V2
with python3. For some period in 2018, both python2 and python3 will
be supportable but only when building local developer packages. LAVA
production releases will stay on python2.7 during testing with
python3. Once we are happy that python3 support is ready, there will
be an email to this list and the following production release will
provide only python3 packages. This change will be transparent for
most instances, the upgrade will simply pull in a lot of python3
packages and the python2 packages will be left unused.
Reviews and contributions to lava-dispatcher V2 already run the unit
tests with python3 and once the V1 code is removed, this will start to
happen with the lava-server code as well.
LAVA can move to python3 at any time once the V1 code is removed - all
the necessary support is already available in Jessie. LAVA will have
to move to python3 before django 2.2LTS is the only version of django
released in Debian stable.
Details of how this will be managed are yet to be defined. We expect
to continue parallel support of python2.7 and python3 for a limited
time.
It is likely that support for jessie-backports would need to cease
before support for both python2.7 and python3 is included, just to
manage to matrix of builds and tests. LAVA can support python3 using
django 1.11LTS but must drop support for python2.7 to support django
2.2LTS.
Summary
=======
17th June 2017 - expected date of Debian Stretch release
August 2017 - migration of Cambridge lab to Debian Stretch instead of Jessie
November/December 2017 - removal of V1 codebase
Q1 2018 - testing with django 1.11 LTS
Q2 2018 - testing with python3, likely end of support for jessie-backports
Q3/4 2018 - add python3 support
Q1 2019 - drop python2.7 support.
April 2019 - release of django 2.2LTS
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
jessie-backports
============
New package update required from jessie-backports.
sudo apt -t jessie-backports install python-voluptuous
This package is used to check the submission schema of V2 test jobs.
It is already installed for V2 support but 2017.5 requires an updated
version which already exists in jessie-backports. For a smooth upgrade
ensure this package is upgraded as above before installing 2017.5.
As with previous releases since 2016.12, the 2017.5 release is only
available from the lava repositories:
https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…
New device-type templates
===================
2017.5 includes a lot of new device-type templates and a number of V1
device-type configurations. These have been contributed to LAVA, as
part of the support for KernelCI, by community members. Many thanks to
all those who have contributed device-type support for this release.
Action names and effects on timeouts
===========================
2017.5 makes the individual names of actions in the source code
consistent across all classes and deployments. Where previously
power_off was available as an action name which could be specified in
timeouts and protocols, that action name now uses a hyphen instead of
the underscore, power-off. The same change has been made for all
classes across the V2 lava-dispatcher codebase. Future changes will
only be accepted using a hyphen to separate words, not underscores or
other characters.
This change affects V2 jinja2 templates as well as V2 test jobs (if
those jobs override the action timeout or call an action using
protocols.
The full list of changes is below:
lava-server
========
8ec2d6736 Fix jinja2 templates for default string handling
eb39ec0fd Extend bbb template check for ssh_host support.
55a251b87 Fix documentation example test job and remove unused
7769be718 Update docs for change in submit behaviour.
c10bf6197 Add more index items and detail on namespaces
b9686e5b1 Mark V1 XML-RPC functions as deprecated.
a0ac9894b Prevent health check warning when disabled
78f796eee more silencing of unit test logging
064bfabb9 Add a template for frdm-kw41z and delete a duplicate for k64f
a08df108d Silence logging in more unit tests
97dbd95b1 Extend power-off timeout for b2260
0e39966d4 device-types: add Hardkernel meson8b-odroidc1 board
b6151b003 Add support for aliases in device-type management
15589b8d1 Fix some typos in development documentation.
f476a351c device-types: base-uboot: use run bootcmd
6911b8502 Expand notes on reviews
cbbdcfcf4 Expand notes on code analysis around reviews
970e52230 Drop confirmation page on job submit for V2 jobs.
4a5050691 Device commands are allowed to be lists
cc60805db Adjust hikey template to allow target_mac and ip support
9d4878c40 Avoid forcing the date path immediately
68497b557 Remove the character_delay block override for d03
5aeed4aef Tags: fix HTML syntax errors
a0ae8a62d Update doc for adding a pipeline worker.
2f6287e71 Add Raspberry pi devices
69a8f40c4 Add collection of Exynos 4 and 5 devices
e31953519 Add more Tegra124 based devices
b843bfaec Add more r-car generation 3 devices
24d21cdb0 Add a note on https repositories and apt-transport-https package.
c3b852798 Expand notes on portability
c57c9f111 Fix doc to explain unprivileged containers and DUT interaction.
2018f705c Show the requested device tags in the job log page
96999537e Fix Action names (use - instead of _)
f0a272b4b Extend recent job support to requested device type
9040de9ca Add XML-RPC call to obtain job level metadata
a04086dd7 api: add get_recent_jobs_for_device
8bbd298f7 Set the documented flash_cmds_order for hikey
c9b9453d1 Migrate many U-boot devices to v2 configuration
613d17fb5 Tweak the developer workflow to skip devices/
c59cc6c31 Add a unit test for some of the new UBoot support.
e43e244b0 Remove unused imports and unused variables.
e2b3784f4 Fix pep8 error
af3cc6d42 Schema: Allow boolean variables in parameters
4d5861607 Allow is_valid check to operate correctly.
294bb62bd ensure device_type is checked
b8ce5ba49 Add a note on developer branches
11725bc3e doc: fix a small gap about test suites
6280b94d7 Add "sd" for removable media
21a5ecf6e Add "command" action to schema and device template
8db43278a Add schema validation for test/monitor/name in job definition.
0a5b1eaa3 tweak gitignore
e3f003f63 templates: remove duplicated blocks
f1331ef2e Exclude retired devices from Device Health table
790b39b9d master: use yaml.CLoader that is way faster
749a1081c Add notes on load balancing different bootloaders
26b822d67 Add note on how pyudev replaces / with _
cc9e4ea2c Allow to override U-boot bootcmd command
9a007665c Fix 500 when output.yaml is invalid
b3c2d162b Make it easier to spot incomplete test jobs
fe6af63a7 Improve job and device schema validation
4a568e261 Fix directory and file permissions
f33661ae4 env: fix comments about default values
fd6fe12b6 Fix scheduling when putting a device into looping
99b35ba79 Export the full lava-server version
da87efaa6 base-uboot.jinja2: add support for append_dtb and use_xip
babaef51d lava_scheduler_app: api: Add pipeline information to get_device_status
abe787872 Add a note on installing lava-dev
2b6ff1dca MASTER_CERT of lava-master should use secret key
d9e7e2c08 add recipients in notifications.yaml
44eb1b75b Move job outputs to sub-dirs based on submit-time
839b3ff19 Create directories with 0o755 by default
9459aae36 lava-master: call job.output_dir to get the path
8c9af4897 Fix description for devices and workers
541478930 Fix health-check tests by testing None and ''
68b33d572 Use job.output_dir whenever possible
d93df5f4f Add a management command to remove old jobs
bfd57121d Move unnecessary constants into base jinja template.
lava-dispatcher
===========
3be27df4 lxc protocol: simplify the tests
a5d9ebad Fix detection of missing ssh_host value in validate
234eef91 Fix calling of protocols after LXC change
477a36b7 tests: make ShellCommand.logger a DummyLogger
62867f49 Use images.validation.linaro.org files for unit tests
1868936c Silence logging in more unit tests
609c783c Fix download action name
b1701978 More tweaks to silence messages from the unit tests
29efa263 Adjust for pep8 checks in jessie
0733f89f Make sure the test_character_delay is used for all
commands in test shell
78d2509f Account for empty environment string.
ea09b19a Take namespace into account when counting test stages
734688c5 Declare namespace of the test suite in results
49a6b92c Drop noisy info log message.
4fd5d4af Drop unused imports and unused variables
678b216c Rework the removable action to allow sd cards
3b42122a Fix bug #2975 after qemu-nfs introduction
7462ea62 Expand LXC support to add devices from all sybsystems
b853865f Fix missing check for u-boot commands parameters
8293693c Replace invalid characters in test_case_id.
a94e0f24 Fix preseed/late_command appending
db03483a Fix Action names (use - instead of _)
9e836e64 Add a Command Action
bbd4ba25 Make the parser stricter about the block names
65b3a563 Allow to pass integers to run_command
b41326bc Raise JobError instead of NotImplementedError
073f3b34 update gitignore
00f5d33f log: limit the length of lines send other zmq
5b2b0746 LAVA-889 Fix handling of multiple test blocks
3bfdcdb6 Add support for use_xip
8df17dd7 Add support for append_dtb
3c110cae Remove unused imports
176a4d49 Add new utility function infrastructure_error_multi_paths
dc008eb5 Export the full version string.
daebfad1 Extend secondary connection fix to support primary
fae0ced5 Move unnecessary constants into base jinja template.
18f745bc device-types: add kirkwood-db-88f6282.conf
815e518d device-types: add at91sam9m10g45ek.conf
6ca0731f device-types: add at91rm9200ek.conf
003b53dc device-types: add at91-sama5d4_xplained.conf
9b2371cf device-types: add armada-3720-db.conf
5a767331 device-types: add armada-xp-gp.conf
8b050938 device-types: add kirkwood-openblocks_a7.conf
f5801d38 device-types: add alpine-v2-evp.conf
19e20276 device-types: add sama5d34ek.conf
49da704f device-types: add armada-385-db-ap.conf
ed7aa6b1 device-types: add armada-370-db.conf
9a104327 device-types: add at91-sama5d2_xplained.conf
ddb34cea device-types: add alpine-db.conf
f7642f58 device-types: add armada-375-db.conf
0d70c7dd device-types: add at91sam9x35ek.conf
a642d451 device-types: add armada-xp-db.conf
2a5ff9e4 device-types: add armada-7040-db.conf
1a439995 device-types: add orion5x-rd88f5182-nas.conf
ad855a62 device-types: add armada-xp-linksys-mamba.conf
58308ca1 device-types: add armada-388-gp.conf
6a2898bf device-types: add armada-370-rd.conf
cd1b4f47 device-types: add armada-398-db.conf
51925fb1 device-types: add sun8i-a83t-allwinner-h8homlet-v2.conf
4976d068 device-types: add sun8i-a33-sinlinx-sina33.conf
681480d5 device-types: add sama5d35ek.conf
db74d60e device-types: add sun5i-r8-chip.conf
40736b95 device-types: add sama5d36ek.conf
433be0a6 device-types: add imx6q-nitrogen6x.conf
ed4c82e6 device-types: add at91sam9x25ek.conf
ce7a488e device-types: add at91sam9261ek.conf
f87cdb49 device-types: add armada-xp-openblocks-ax3-4.conf
634442ca device-types: add armada-388-clearfog.conf
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
