The LAVA software team are happy to announce the roadmap for the removal of
LAVA V1 support, test data and source code.
The next LAVA production release will be 2017.9 which will be the last
release with functioning V1 support. There are no changes to V1 in 2017.9,
so it will continue working as it does currently for this release.
https://projects.linaro.org/browse/LAVA-994 tracks the work for 2017.9
Any instance where V1 test jobs need to continue running must NOT upgrade
beyond 2017.9. There will be no future updates of V1 for any reason and no
support for anything related to V1 other than to migrate V1 test jobs and
devices to V2.
Blocking V1 test job submissions with 2017.10
=============================================
The 2017.10 release will *disable* the submission of all V1 test jobs
either from the UI or using XML-RPC or lava-tool. If any devices still have
V1 health checks, those health checks will no longer run. Any queued V1
test jobs will run. Only V2 test jobs will be allowed to be submitted.
https://projects.linaro.org/browse/LAVA-1026 tracks the work for 2017.10
If you want to keep a read-only archive of your V1 test data, then 2017.10
is the last chance to create this archive. Any instance where the V1 data
should remain visible must NOT upgrade beyond 2017.10.
Removing access to V1 URLs in 2017.11
=====================================
All URLs related to V1 will be removed or disabled in the 2017.11 release -
the Dashboard drop-down menu will disappear and any bookmarks to dashboard
URLs would give a 404. The Dashboard XML-RPC API will become no-ops
returning no data. This means that bundles and bundle streams will not be
able to be downloaded or viewed. Filters, Image Reports and Image Reports
2.0 will all become unavailable.
V1 documentation will be deleted in this release - the V2 docs will be
promoted as the LAVA documentation, removing references to V1.
The V1 source code for lava-dispatcher will be deleted at this point. There
will be some changes to the directory structure of the remaining code, so
if you are maintaining out of tree patches, you will need to fix some merge
conflicts.
Elements of the web UI which distinguish between V1 and V2 support will
also be disabled and removed. This will include removing access to viewing
or downloading the log files of V1 test jobs.
https://projects.linaro.org/browse/LAVA-1028 tracks the work for 2017.11
DELETION of V1 test data in 2017.12
===================================
This has been highlighted many times in LAVA announcements, Connect
sessions, in the documentation and in all support enquiries. It is finally
happening with 2017.12
Any instance which installs 2017.12 will directly cause the deliberate and
permanent deletion of ALL V1 TEST DATA for that instance. Bundles,
BundleStreams, Filters, Image Reports, Image Reports 2.0 - everything in
the database which relates to V1 will be deleted.
Alongside the removal of the test data, the V1 source code will also be
deleted from the lava-server codebase.
*This is a one-way process*. If there is no usable backup for the instance,
there will be NO way to restore any V1 data after this point. After
restoring from backup, if 2017.12 or any later release is installed, the V1
data will again be deleted.
https://projects.linaro.org/browse/LAVA-1029 tracks the work for 2017.12
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
There are delays getting packages into stretch-backports, as expected.
In the meantime, this is a reminder of how to use backports: first
start with stable.
So when installing LAVA on Stretch, even if what you want is the
latest release from production-repo or staging-repo (currently
2017.7), then your first step is to install 2016.12 from Stretch.
# apt -y install lava-server
# apt -y install vim
# wget http://images.validation.linaro.org/production-repo/production-repo.key.asc
# apt-key add production-repo.key.asc
# vim /etc/apt/sources.list.d/lava.list
# # edit the file to specify: deb
http://images.validation.linaro.org/production-repo jessie-backports
main
Yes, that is jessie-backports - we don't have packages in
stretch-backports at this time.
# apt update
# apt upgrade
If you specify backports too early, you'll get a range of other
packages from backports - you don't actually need jessie-backports or
stretch-backports from Debian at this time.
Packages for jessie-backports are built on jessie. Packages for
stretch-backports are built on stretch. It's the same source code in
each case. Right now, there aren't any problems with installing from
jessie-backports on stretch - particularly if you install lava-server
from stretch in the first place so that the majority of your packages
come from stretch.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
During testing, we've seen that upgrades to Stretch from Jessie can
have an issue if UWSGI is still left from a previous installation:
apache2: Syntax error on line 146 of /etc/apache2/apache2.conf: Could
not open configuration file /etc/apache2/mods-enabled/uwsgi.load: No
such file or directory
Action 'restart' failed.
The Apache error log may have more information.
dpkg: error processing package lava-server (--install):
subprocess installed post-installation script returned error exit status 1
Setting up lava-server-doc (2017.7-1~bpo9+1) ...
Setting up lava-dev (2017.7-1~bpo9+1) ...
Processing triggers for systemd (232-25) ...
Processing triggers for man-db (2.7.6.1-2) ...
Errors were encountered while processing:
lava-server
The problem is because of a bug in the uwsgi packages. The uwsgi
module is not cleanly removed from apache. Run this command to fix:
$ sudo a2dismod uwsgi
Then complete the install:
$ sudo apt-get -f install
Fresh Stretch installs should not be affected as 2016.12 in Stretch
does not use uwsgi.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
The 2017.7 release is available from the lava-repositories for
jessie-backports and has been uploaded to Debian unstable. Support for
stretch-backports is pending. (The production-repo/jessie-backports
can be used in the meantime.)
https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…
jessie-backports "sloppy"
==================
This release has been uploaded to Debian unstable. The period after a
Debian stable release is turbulent in unstable and the LAVA packages
may not migrate into buster (testing) smoothly which will delay
uploads to the official backports archive (for jessie backports and
stretch backports).
To update jessie-backports, the upload will need to be the version
from buster, rebuilt for jessie and sent to jessie-backports-sloppy.
Once that is done, I'll post again to the this list about how that
becomes available to admins running Jessie systems.
Jessie continues to be the preferred release for LAVA until such time
as we've done more testing on Stretch. We don't anticipate any
problems with lava-server or lava-dispatcher on Stretch as this has
been tested frequently. However, third party packages and scripts
still need testing with real instances using Stretch.
V1 support in buster
===============
Some unit tests in the V1 support have had to be disabled due to bug
fixes in django 1.11 LTS in buster and sid affecting the atomic
transaction handling. The remote worker design in V1 has always had
issues with atomicity and these unit tests have failed intermittently
for a while. This is one of the many reasons to drive all instances to
V2. These tests are now failing consistently with 1.11, so have been
disabled if django 1.11 or newer is detected.
V1 is no longer run on any of our development instances (running
buster or sid), only in production instances (running jessie) which
are in preparation for the migration. We can only assume that the
failure of the unit tests would make V1 unreliable on buster or sid.
However, with the pending removal of V1, there will not be any fixes
for any problems discovered when using V1 with django 1.11 or newer.
LDAP support in buster and sid
=======================
python-django-auth-ldap has changed dependencies, from python-ldap to
python-pyldap. When upgrading python-django-auth-ldap, python-ldap
will be removed and python-pyldap installed.
Instances *must* restart the lava-server-gunicorn service after this
change or users will experience HTTP 500 errors.
lava-server manage check --deploy
=========================
This release adds some checks to the django system checks. When admins
update health checks or device-type templates, it is worth running the
deployment check which does a quick sanity check on the instance. This
can help to spot invalid health check submissions amongst other
issues.
Note that on development instances, some issues will always be
reported - e.g. when Debug is set to True for django using
/etc/lava-server/settings.conf
Also, for developers, some of the improved security features in django
may get in the way of signing in to http://localhost by enforcing
requirements for https:// which is unusual on localhost.
Check the django documentation - to enable localhost, you may need to
disable at least these security defaults by adding to
/etc/lava-server/settings.conf:
"CSRF_COOKIE_SECURE": false,
"SESSION_COOKIE_SECURE": false,
As a result of these changes, installing 2017.7 can generate a prompt
to update /etc/lava-server/settings.conf. In general, keep your
existing configuration but look at the differences and migrate the
changes across. Remember to restart lava-server, lava-master and
lava-server-gunicorn services whenever /etc/lava-server/settings.conf
is modified.
Skipping 2017.8
============
There won't be a full 2017.8 production release of lava-server or
lava-dispatcher as a number of the team are away at the annual Debian
conference, DebConf17 in Montreal.
https://debconf17.debconf.org/
If you are going to be there, let us know.
lava-server changes
==============
a9390fc9a Remove debug hacking session from doc examples
13edd8f12 Skip certain V1 unit tests on django 1.11 and later
0577447fc Update query script for python3 and to output CSV
0b3b0beb7 device-type: addresses fix for armada-3720-espressobin
ed1e402e5 device-type: addresses fix for armada-3720-db
c382a5aa0 device-type: fix at91sam9m10g45ek
54a3b7a46 Apply change for distinct test case links to all results
1824a18fb Adding uefi-menu tftp method
71aac817e LAVA-363 - set the scheme in api/help from settings
2c840debc base-uboot.jinja2: don't load DTB if append_dtb is set
1b13a0e3e Fix extra keyword in api/help view
2882529bc Provide an example script for using queries
7f4cfe569 logs: continue to poll the log when canceling
25eaab176 XMLRPC: add offline_info argument to list endpoint
38f7570e3 XMLRPC: fix inconsistency between code and doc
8c134839d LAVA-972 test case should use ID in URL.
99725606c Remove references to OpenID and Crowd
95b959602 XMLRPC: add a function to download job definition
ebf80c266 XMLRPC: fix crash when device.worker_host is None
b1ec14c89 LAVA-861 separate base-fastboot.jinja2 template
c4fd79d57 Remove unused file
7c066195a Remove deprecated django configuration
520ac8778 XMLRPC: remove unused function
3b8ab7d61 XMLRPC: do not raise an error when canceling twice
cce1bdea6 XMLRPC: allow to force health check for a device
218408db6 Handle the new log level called "input"
b49d9fb96 Adding uefi parameters for vexpress
4a52ba861 Allow overriding django system check values from settings as
documented
d2067a5f3 LAVA-987 fix log parsing
bbc39489c results_app: API: add the 'run_query' function
8746edb6b results_app: API: add the 'make_custom_query' function
4fd429ac2 Add note on apache2 for V2-only workers
641e61dcb Explain the handling of measurements and Decimal
bf8f53bf5 Count going offline as busy in overview table
06febd188 Expand on essential roles, results and sync
815d41b52 Tighten advice on prompt strings in docs
351f6fe41 Add documentation for login_commands in auto_login
5a7d4be95 Avoid warning for health checks of retired devices.
5713f5295 Document transfer_overlay deployment differences
8f04c6097 Fix typo in multinode api documentation.
fe2093994 device-types: add meson-gxl-s905x-khadas-vim
853ee80e3 Move action fields templates to table files.
12de946eb Allow bare except and override pep8
dd4faacaf Enforce schema rules for metadata
352049204 Add missing check for null key|value in metadata
b77731b31 device-types: rcar-gen3-common: handle bigger kernels
d2acb94a5 LAVA-983 - Helper for DeviceStateTransition
b4208116c LAVA-980 - Fix django deployment warnings
778fd4a8b Fix link to glossary term
af85e5225 LAVA 931 - documentation.
212e98a3a LAVA-976 Use the Django system check framework
e9c7ad477 Adding commands for booting whatever the precanned defaults are
f78c58811 Added device type for Cavium ThunderX
lava-dispatcher changes
=================
d8dabe63 Switch the print to a test in unit test
d1e22b67 Revert "Replace invalid characters in test_case_id."
b3b4f5ac Update the example to a working test job with prompts.
504ec2bc LAVA-777 last log line should be lava.job result
ad55acb9 Ensure overlay unpacking is clearly logged
2e7c9598 Add pipeline_ref update for test skipped by lavabot
ed936cd3 Simplify the power control actions
78dcc1b0 Fix dispatcher configuration
89880ce1 LAVA-96 Check for errors if a tftp file exceeds 4G
6fd2702e LAVA-989 add extra logging for actions
df86ed30 LAVA-990 add logging for VLANd + MultiNode support
bbfb1817 Stop the unit test outputting noise
c7839a98 Add a new log level for the DUT inputs
2502b8af Destroy LXC containers properly.
957734cf Add a unit test for Vland actions
68468abe Handle a bad url cleanly.
713d4e6e LAVA-645 - summarise slave logging
96a5effe Do not raise exceptions in validate
f45106fb Catch errors in testdef from handler cleanly
f34f41e2 LAVA-984 - LXC protocol timeout to report job result
e31722b5 Allow ci-run with updated pep8
175d8e84 LAVA-931 - MultiNode lava-sync should report results
fd944543 EOF in a test shell is an InfrastructureError
429691d9 Support power control for CMSIS devices
049c0227 lava-slave: add the right configuration for the host
1e92bf38 lxc: improve error message
b2f25cf0 Add support for booting from UEFI shell
67a1e1c6 ramdisk: only pass to bootx if has u-boot header
f4a4abd9 LAVA-967: check namespaces usage
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
As with previous releases since 2016.12, the 2017.6 release is only
available from the lava repositories:
https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…
However, Debian Stretch is due to be released very soon. 2017.6 may be
the first release to be uploaded to Debian unstable since the stretch
freeze started with 2016.12. There may be a short delay whilst
unstable settles down after the usual round of extra uploads. Once
packages migrate into the new testing (buster) and stretch-backports
becomes available, packages will be uploaded for stretch-backports and
jessie-backports.
Device dictionaries moved to the filesystem
===========================================
LAVA-757 - all existing device dictionaries will be migrated to the
filesystem (/etc/lava-server/dispatcher-config/devices/) during the
installation of the 2017.6 production release. The dependency on
python-django-kvstore has been dropped. This means that there is no
longer a need to update the device dictionary in the database, simply
install the new file into the directory and the next test job to be
assigned to that device will use the new configuration. Unlike
device-types, there are no default device dictionaries provided by
LAVA, so a fresh install will have no device dictionary files in
/etc/lava-server/dispatcher-config/devices/
The documentation has been updated. There is now no need for
lava-server manage device-dictionary, so this command has been
dropped. Instead, admins who have permissions to run lava-server
manage can simply copy a device-dictionary .jinja2 file into
/etc/lava-server/dispatcher-config/devices
During installation, this migration will take place:
Applying lava_scheduler_app.0027_device_dict_onto_filesystem
Database entries for all device dictionaries are replaced with files.
If a device can support V2 but has no device dictionary, it will be
marked as Skipped.
This change means that lava-server no longer depends on the
python-django-kvstore package.
Feedback support with namespaces
================================
Devices may output messages to the serial console as a result of
operations performed within an LXC and this output is now logged as
part of the test job log file as 'feedback'. See the documentation for
more information on feedback output.
Architecture is optional for LXC
================================
Some LXC deployments support identifying the deployment architecture
based on the host system. Some instances may use workers of different
architectures and test jobs should be portable between instances.
Therefore, the arch: field of the lava-lxc protocol has been made
optional. Note that Fedora and Ubuntu deployments still need the
architecture to be specified for x86 machines to distinguish between
32bit and 64bit x86 deployments.
The architecture value can still be specified in test jobs but the
test jobs will be more portable if omitted.
Action names use hyphens, not underscore
========================================
If you are using action-specific timeouts in your test jobs or in
customised device-type templates, note that the names of some actions
have *changed*. This is to make the usage of hyphens and underscores
consistent. Actions with names containing underscores are now invalid
and all underscores in names have been replaced with hyphens.
'pdu_reboot' becomes 'pdu-reboot'.
lava-server changes
===================
daf1e2a lava-master: fix init script
b6b5fe7 Reduce device configuration dynamic connections
ebf4463 Fix a crash in the migration when the db is empty
cdfeb12 Fix index out of range exception
33092ef Adding usb deployment method to vexpress device template
3257ee2 XML-RPC: remove unused imports
55dd014 XML-RPC: fix crash when called by anonymous users
a3d4718 XML-RPC: allow every user to get templates/dict
ddaf693 Add documentation of feedback support.
eaae89a Set a default connection timeout for lava-test-shell
f305e47 publisher: fix umask
c25686f logrotate: lava-master should be owned by lavaserver
2da8e74 logs: except more exceptions
14f5f6c Update docs for changes in 2017.6
a486d12 Tidy up of debian support docs
b05712d Preserve original comments when resubmitting
19bd2db XML-RPC: add an api to manage aliases
30973a0 Log an error when saving device configuration fails
763239d Use --arch for fedora LXC example
8f26102 XML-RPC: add a method to show tag details
19a69f7 fix location of example test job files
480a6f6 XML-RPC: add an api for handling jobs
356667d Update installation docs for Stretch release
da74ae5 XML-RPC: add a decorator for is_superuser check
49f370c XML-RPC: add a clean api for device-types and aliases
c6d503d commands/devices: extend devices command to add tags
1655dbf Move set commands outside body block
87b8c01 XML-RPC: add a clean api for workers and tags
351c16b XML-RPC: add a clean api for devices
c4dec1f device-types: add bcm2837-rpi-3-b
9ca0e0d Adding combined vexpress device template
67f531f Make lava_scheduler_app.api a module
d3021a3 xmlrpc: add a system.version function
3ef0157 linaro_django_xmlrpc: remove unused properties
e994852 Use render shortcut instead of the loader
e51ca21 linaro_django_xmlrpc: rework the application
4bbe17c LAVA-959 run lava-master as lavaserver user
b283288 Fix migrations from Debian Jessie to Stretch and ensure smooth upgrade.
bc25204 LAVA 329 - Add old job URL to metadata when resubmitting a job.
271723d LAVA-782 - Change error type for incorrect handling of custom queries.
368903e notify: Don't send verbose mails when the job is cancelled
71c514d Fix broken link in unit test.
1f48674 LAVA-955 - Remove 'arch' parameter from lxc protocol.
3b36ba0 Add auto_login test case in test_pipeline unit test
e23bb17 Fix typo which causes XML-RPC error when saving device dict
9a82aee XML-RPC: allow to have sub_modules
b0c7a5b Document reboot and apply-overlay elements for fastboot deploy method.
8a93049 LAVA-935 - Provide feedback output from connections
d71f267 Fix typos in proxy configuration documentation.
c7227b3 Fix rebase error in grub hikey test
9253497 Only the master should create log directories
1ee736b Allow every u-boot devices to use minimal boot
e816c17 LAVA-832 allow to manage users from the command line
eb83cbe Add grub interrupt prompt and interrupt character to constants
ee79de2 LAVA-937 show job visibility and fix group visibility
73b6d86 Prevent key error in metadata handling.
f89e840 Improve job output migration helper
eb1ad02 device dict: do not show exclusive flag
3900276 Restore date-based subdirectories
1faa9f0 LAVA-855 - ART CI: Intel NUC device integration for LAVA v2
d195b01 Allow to revert (dummy) the migration
35f0802 Fix device dict export
da7ebe2 device dict: fix web rendering
4a3daa9 Allow override of root for installed mustang.
12c0508 Expand zmq_client to handle publisher socket
34f3625 Trivial whitespace changes in glossary
6246a5d Improve data export docs
2af8e72 Allow for non-Const values in device dictionary
59742fe Extend examples to use the DNS support for NFS
c514296 Fix change in import paths for V1 and V2 scheduling
765901e Add notes on advanced features for submission
fd0905c Remove deprecated commands
0237b85 Use CommandError whenever possible
a1cc08b Improve device-type templates
05123cc Missing conditional in device-type alias handling
17af14e Move commands to lava_server app
5bab54d Remove reference to unavailable boot method 'fastboot-boot-image'.
78e9048 Add mediatek dt for v2
49f27fa Allow in-place YAML include in V2 job submission.
c2be83f Grub support for HiKey
ae4d477 LAVA-757 Move device dictionaries to file system
52eb1e1 Remove unused model JobPipeline
9d0a5ae results: make the link unique
55bf39f Add a callback_url functionality to notification schema.
0324915 Add login_commands to the auto_login schema
ce6d0f7 Don't reload template from disk on each access
lava-dispatcher changes
=======================
38da5d8 LAVA-974: Fix init script and allow to set hostname
0fd687e Don't use os.path.realpath() on device names
d84dab7 Modifying vemsd deploy action to use ResetDevice
d21385e testdef: get 'revision' from the right variable
5ea16fe Rework test shell feedback timeouts
510c4e5 Declare the feedback timeout
b994757 Use pyudev device links as well as device nodes
e609c22 Fix namespace usage in power commands.
39e2e7e Read from feedback connection during long operations
7be926a Allow tar.gz firmware bundles
20e66c1 overlay: Raise JobError if root_partition is None
4e27fcd LAVA-935 - Provide feedback output from connections
831c71c Fix bug #3032 - Running an LXC/Hikey job without a test action ...
84e7378 Fix namespace issue in NFS deployment action
f1f8d71 Add login_prompt valid test case
7dc48aa Issue lxc-stop before destroying a container.
3e1cb07 Changing udev checks to only look for FS label
d08db39 Fix #3010: set logging timezone to UTC
1b664e5 LAVA-955 - Remove 'arch' parameter from lxc protocol.
8a4dbc7 Increase LXC finalize timeout to 90 seconds similar to multinode.
3823c49 Use one debug log call to log the same kind of message in lxc protocol.
5db313c Creating vemsd mount points
ea03ef1 Fixing boot test issues using NFS deploy action
d1f4a30 Fix rebase error with grub prompt constant
b2aef8b device-types: add sun6i-a31-app4-evb1
8ceca07 Make grub use interrupt prompt and character from constants
b495a35 device-types: add ttyAT0 to Atmel devices to support multiv7_defconfig
3a89027 device-types: fix at91sam9x25ek to use bootz
9e00895 Set download max_retries in the constructor
959480c Add minimal boot action
31fd021 User command: do not clean if it was not executed
26572a2 Fix keys and label that were outdated
67dd60e Remove unused functions
0c100ff Remove boot-result common data
342039a Fix RetryAction inside a RetryAction
9ce430f Fix check on LXC for default usage.
0576754 Fix typo: Use pixel job create function for testing pixel device.
e7a4422 Fix typo: remove visibility added twice.
9e67b6d LAVA-855 - ART CI: Intel NUC device integration for LAVA v2
dd5d874 Ensure all deployments can add LXC devices except QEMU
e72d490 Fix pipeline refs for change in monitor retry
cb57d2b LAVA-942 UEFI Menu and command lists
e57013b Start adb daemon before attempting adb commands.
ac389ad lava-slave: remove tmp dir when END_OK is received
632a0ff Fix device configuration
0cffce4 Fix validation of boot auto_login parameters and add a unit test
7e40701 Allow unix line separators for some UEFI devices
095798a Avoid calling cleanup in validate
f98dade Allow updating all pipeline_refs at a time.
31b62dd LAVA-928 - HiKey issue when switching between AOSP and OE
51f40ef Make soft reboot optional LAVA-846
4d43d02 HiKey Grub EFI support
874f889 Avoid logging the same message twice.
3be27df lxc protocol: simplify the tests
75126ff Do not set self.errors when Action.run fail
ce32bfb Do not use self.errors in Action.run()
a5d9eba Fix detection of missing ssh_host value in validate
234eef9 Fix calling of protocols after LXC change
477a36b tests: make ShellCommand.logger a DummyLogger
62867f4 Use images.validation.linaro.org files for unit tests
1868936 Silence logging in more unit tests
609c783 Fix download action name
b170197 More tweaks to silence messages from the unit tests
29efa26 Adjust for pep8 checks in jessie
0733f89 Make sure the test_character_delay is used for all commands in
test shell
78d2509 Account for empty environment string.
ea09b19 Take namespace into account when counting test stages
734688c Declare namespace of the test suite in results
60561c2 Parser: remove unused context
49a6b92 Drop noisy info log message.
8ed8467 lava-test-monitor: use TestMonitorRetry action
4fd5d4a Drop unused imports and unused variables
678b216 Rework the removable action to allow sd cards
3b42122 Fix bug #2975 after qemu-nfs introduction
7462ea6 Expand LXC support to add devices from all sybsystems
b853865 Fix missing check for u-boot commands parameters
8293693 Replace invalid characters in test_case_id.
a94e0f2 Fix preseed/late_command appending
9647d55 Run login_commands if provided in auto_login
db03483 Fix Action names (use - instead of _)
9e836e6 Add a Command Action
bbd4ba2 Make the parser stricter about the block names
65b3a56 Allow to pass integers to run_command
b41326b Raise JobError instead of NotImplementedError
073f3b3 update gitignore
00f5d33 log: limit the length of lines send other zmq
3f99cc3 Add missing super call to WaitUSBDeviceAction
5b2b074 LAVA-889 Fix handling of multiple test blocks
5436658 Fix indent typo
3bfdcdb Add support for use_xip
8df17dd Add support for append_dtb
3c110ca Remove unused imports
4847bf1 Put device rebooting info message at the correct place.
6b85382 Fix call to lava-lxc protocol for pre-os-command
176a4d4 Add new utility function infrastructure_error_multi_paths
bdb4d12 Fix test_lxc_api unit test
dc008eb Export the full version string.
e9df320 LAVA-920 - Workaround ptable issue in firmware
7b4096c Allow LXC protocol to call pre-os-command
b33d04b Fix OE image support for HiKey.
d714a9b Add missing test shell helpers
daebfad Extend secondary connection fix to support primary
9518e74 Add missing import
e97801d Add optional deployment complete message list to secondary
media deployments
fae0ced Move unnecessary constants into base jinja template.
18f745b device-types: add kirkwood-db-88f6282.conf
815e518 device-types: add at91sam9m10g45ek.conf
6ca0731 device-types: add at91rm9200ek.conf
003b53d device-types: add at91-sama5d4_xplained.conf
9b2371c device-types: add armada-3720-db.conf
5a76733 device-types: add armada-xp-gp.conf
8b05093 device-types: add kirkwood-openblocks_a7.conf
f5801d3 device-types: add alpine-v2-evp.conf
19e2027 device-types: add sama5d34ek.conf
49da704 device-types: add armada-385-db-ap.conf
ed7aa6b device-types: add armada-370-db.conf
9a10432 device-types: add at91-sama5d2_xplained.conf
ddb34ce device-types: add alpine-db.conf
f7642f5 device-types: add armada-375-db.conf
0d70c7d device-types: add at91sam9x35ek.conf
a642d45 device-types: add armada-xp-db.conf
2a5ff9e device-types: add armada-7040-db.conf
1a43999 device-types: add orion5x-rd88f5182-nas.conf
ad855a6 device-types: add armada-xp-linksys-mamba.conf
58308ca device-types: add armada-388-gp.conf
6a2898b device-types: add armada-370-rd.conf
cd1b4f4 device-types: add armada-398-db.conf
51925fb device-types: add sun8i-a83t-allwinner-h8homlet-v2.conf
4976d06 device-types: add sun8i-a33-sinlinx-sina33.conf
681480d device-types: add sama5d35ek.conf
db74d60 device-types: add sun5i-r8-chip.conf
40736b9 device-types: add sama5d36ek.conf
433be0a device-types: add imx6q-nitrogen6x.conf
ed4c82e device-types: add at91sam9x25ek.conf
ce7a488 device-types: add at91sam9261ek.conf
f87cdb4 device-types: add armada-xp-openblocks-ax3-4.conf
634442c device-types: add armada-388-clearfog.conf
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
This is a long mail but there is a lot to cover...
Stretch
======
Debian Stretch is expected to be released on June 17 2017:
https://lists.debian.org/debian-devel-announce/2017/05/msg00002.html
As has been mentioned on this list before - admins MUST check the apt
sources for all machines *NOW* to ensure that all your Debian apt
sources are pointing at 'jessie' NOT 'stable' if you are to be able to
manage the upgrade yourself.
What you want in /etc/apt/sources.list and /etc/apt/sources.list.d/* is:
deb http://mirror.bytemark.co.uk/debian jessie main
deb http://mirror.bytemark.co.uk/debian jessie-backports main
(The mirror you choose is up to you.)
If you find any apt sources referencing 'stable' or 'testing' CHANGE
THESE NOW or those systems could automatically upgrade to Stretch or
Buster respectively on 17th June.
As with all such releases, production machines are recommended to
allow some time for final testing before doing the upgrade. The
Cambridge LAVA lab is expected to upgrade from Jessie to Stretch
during August 2017. The LAVA software team have been doing testing of
this upgrade for some time, without issues. However, real machines
with extra packages and local admin configuration cannot be tested in
this way, so each lab should take some time to plan when and how to do
the upgrade. (Hint: have a backup and check that it can be restored
correctly before starting.)
https://validation.linaro.org/static/docs/v2/admin-backups.html
There has been a fix for lava-server in stretch to smooth the upgrade
for those users who have not installed lava-server from
jessie-backports. Those already running 2017.* are not affected by
this change and can continue using jessie-backports and the LAVA
repositories as before. Stretch will release with lava-server
2016.12-2 and python-django 1:1.10.7-2
Once Stretch is released, Debian testing will be called 'buster',
Debian stable will be 'stretch' and Debian oldstable will be 'jessie'.
At a later date, jessie-backports will be opened again and a new
stretch-backports will become available. New LAVA production releases
will be uploaded to unstable to migrate into testing (buster) and then
be backported to stretch-backports and jessie-backports. The LAVA
repositories will be updated as well.
Support for jessie-backports will continue after stretch-backports
support is added but cannot continue indefinitely. At some point, all
instances will need to have moved to Stretch to continue getting
updates of LAVA. (Prior to the process of releasing Debian Buster, the
jessie packages will be dropped from the archive, including
jessie-backports.)
Changes in LAVA repositories
=======================
2017.6 will be released in the same way as previous releases in 2017 -
to the LAVA repository for use on jessie using jessie-backports.
At a later point, 2017.6 will be uploaded to Debian unstable for
migration into testing (buster). Only developers should install that
version as Debian unstable can be problematic in the first few weeks
after a Debian stable release.
There are changes in how the lava packages are built on jessie versus
on stretch, so the lava repositories will be split. I'll be updating
the documentation in 2017.6 to cover the changes required to help
admins use the new suites. There will be a new jessie-backports suite
and a new stretch-backports suite. The existing 'sid' or unstable
suite will have no uploads after 2017.6.
This will be covered again in the 2017.6 release email and again with
2017.7. For now, no changes are required.
Once the new suites are announced, admins who choose to upgrade to
Stretch would use the stretch-backports suite from the LAVA
repository.
deb https://images.validation.linaro.org/production-repo stretch-backports main
Those who will upgrade later, would use jessie-backports from the LAVA
repository.
deb https://images.validation.linaro.org/production-repo jessie-backports main
The purpose of these suites is to allow admins to choose to upgrade to
the latest production release for jessie or stretch without waiting
for the migration of the packages into Debian testing and the
subsequent uploads to backports. The packages are otherwise the same.
Django
======
Longer term, there have been a lot of discussions within Debian about
managing django releases. The plan is that only django LTS releases
will be available in Debian unstable, testing and therefore stable
after the Stretch release. Non-LTS releases will only be available in
Debian experimental. This means that lava-server will migrate from one
django LTS to the next django LTS with future Debian releases. The
LAVA software team will manage how deprecation cycles within django
affect lava-server.
This does have the issue that the version of django in Debian stable
will be towards the end of the upstream support cycle but it is better
than having a non LTS django release in Debian stable. Synchronising
releases across multiple upstream teams is not feasible.
The next django LTS release after 1.8 is 1.11 which is expected to be
in the next Debian stable release - buster.
https://www.djangoproject.com/download/
Python3
========
The last django LTS release to support python2.7 will be 1.11 LTS
which retains upstream security and data loss bug fix support until at
least April 2020.
The LAVA software team have already been testing lava-dispatcher V2
with python3. Once the V1 code is removed from lava-server, testing
will proceed for lava-server too. We fully expect to be able to
support lava-dispatcher V2 and lava-server V2 on python2.7 and python3
for a limited time.
There will never be python3 support in lava-dispatcher V1 or lava-server V1.
Python2.7 support for lava-dispatcher V2 and lava-server V2 will be
removed at some point prior to the release of the next django LTS
after 1.11 which is 2.2 LTS, due in April 2019.
Once django 2.0 (non-LTS) arrives in Debian experimental, the LAVA
software team will begin testing lava-dispatcher V2 and lava-server V2
with python3. For some period in 2018, both python2 and python3 will
be supportable but only when building local developer packages. LAVA
production releases will stay on python2.7 during testing with
python3. Once we are happy that python3 support is ready, there will
be an email to this list and the following production release will
provide only python3 packages. This change will be transparent for
most instances, the upgrade will simply pull in a lot of python3
packages and the python2 packages will be left unused.
Reviews and contributions to lava-dispatcher V2 already run the unit
tests with python3 and once the V1 code is removed, this will start to
happen with the lava-server code as well.
LAVA can move to python3 at any time once the V1 code is removed - all
the necessary support is already available in Jessie. LAVA will have
to move to python3 before django 2.2LTS is the only version of django
released in Debian stable.
Details of how this will be managed are yet to be defined. We expect
to continue parallel support of python2.7 and python3 for a limited
time.
It is likely that support for jessie-backports would need to cease
before support for both python2.7 and python3 is included, just to
manage to matrix of builds and tests. LAVA can support python3 using
django 1.11LTS but must drop support for python2.7 to support django
2.2LTS.
Summary
=======
17th June 2017 - expected date of Debian Stretch release
August 2017 - migration of Cambridge lab to Debian Stretch instead of Jessie
November/December 2017 - removal of V1 codebase
Q1 2018 - testing with django 1.11 LTS
Q2 2018 - testing with python3, likely end of support for jessie-backports
Q3/4 2018 - add python3 support
Q1 2019 - drop python2.7 support.
April 2019 - release of django 2.2LTS
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
jessie-backports
============
New package update required from jessie-backports.
sudo apt -t jessie-backports install python-voluptuous
This package is used to check the submission schema of V2 test jobs.
It is already installed for V2 support but 2017.5 requires an updated
version which already exists in jessie-backports. For a smooth upgrade
ensure this package is upgraded as above before installing 2017.5.
As with previous releases since 2016.12, the 2017.5 release is only
available from the lava repositories:
https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…
New device-type templates
===================
2017.5 includes a lot of new device-type templates and a number of V1
device-type configurations. These have been contributed to LAVA, as
part of the support for KernelCI, by community members. Many thanks to
all those who have contributed device-type support for this release.
Action names and effects on timeouts
===========================
2017.5 makes the individual names of actions in the source code
consistent across all classes and deployments. Where previously
power_off was available as an action name which could be specified in
timeouts and protocols, that action name now uses a hyphen instead of
the underscore, power-off. The same change has been made for all
classes across the V2 lava-dispatcher codebase. Future changes will
only be accepted using a hyphen to separate words, not underscores or
other characters.
This change affects V2 jinja2 templates as well as V2 test jobs (if
those jobs override the action timeout or call an action using
protocols.
The full list of changes is below:
lava-server
========
8ec2d6736 Fix jinja2 templates for default string handling
eb39ec0fd Extend bbb template check for ssh_host support.
55a251b87 Fix documentation example test job and remove unused
7769be718 Update docs for change in submit behaviour.
c10bf6197 Add more index items and detail on namespaces
b9686e5b1 Mark V1 XML-RPC functions as deprecated.
a0ac9894b Prevent health check warning when disabled
78f796eee more silencing of unit test logging
064bfabb9 Add a template for frdm-kw41z and delete a duplicate for k64f
a08df108d Silence logging in more unit tests
97dbd95b1 Extend power-off timeout for b2260
0e39966d4 device-types: add Hardkernel meson8b-odroidc1 board
b6151b003 Add support for aliases in device-type management
15589b8d1 Fix some typos in development documentation.
f476a351c device-types: base-uboot: use run bootcmd
6911b8502 Expand notes on reviews
cbbdcfcf4 Expand notes on code analysis around reviews
970e52230 Drop confirmation page on job submit for V2 jobs.
4a5050691 Device commands are allowed to be lists
cc60805db Adjust hikey template to allow target_mac and ip support
9d4878c40 Avoid forcing the date path immediately
68497b557 Remove the character_delay block override for d03
5aeed4aef Tags: fix HTML syntax errors
a0ae8a62d Update doc for adding a pipeline worker.
2f6287e71 Add Raspberry pi devices
69a8f40c4 Add collection of Exynos 4 and 5 devices
e31953519 Add more Tegra124 based devices
b843bfaec Add more r-car generation 3 devices
24d21cdb0 Add a note on https repositories and apt-transport-https package.
c3b852798 Expand notes on portability
c57c9f111 Fix doc to explain unprivileged containers and DUT interaction.
2018f705c Show the requested device tags in the job log page
96999537e Fix Action names (use - instead of _)
f0a272b4b Extend recent job support to requested device type
9040de9ca Add XML-RPC call to obtain job level metadata
a04086dd7 api: add get_recent_jobs_for_device
8bbd298f7 Set the documented flash_cmds_order for hikey
c9b9453d1 Migrate many U-boot devices to v2 configuration
613d17fb5 Tweak the developer workflow to skip devices/
c59cc6c31 Add a unit test for some of the new UBoot support.
e43e244b0 Remove unused imports and unused variables.
e2b3784f4 Fix pep8 error
af3cc6d42 Schema: Allow boolean variables in parameters
4d5861607 Allow is_valid check to operate correctly.
294bb62bd ensure device_type is checked
b8ce5ba49 Add a note on developer branches
11725bc3e doc: fix a small gap about test suites
6280b94d7 Add "sd" for removable media
21a5ecf6e Add "command" action to schema and device template
8db43278a Add schema validation for test/monitor/name in job definition.
0a5b1eaa3 tweak gitignore
e3f003f63 templates: remove duplicated blocks
f1331ef2e Exclude retired devices from Device Health table
790b39b9d master: use yaml.CLoader that is way faster
749a1081c Add notes on load balancing different bootloaders
26b822d67 Add note on how pyudev replaces / with _
cc9e4ea2c Allow to override U-boot bootcmd command
9a007665c Fix 500 when output.yaml is invalid
b3c2d162b Make it easier to spot incomplete test jobs
fe6af63a7 Improve job and device schema validation
4a568e261 Fix directory and file permissions
f33661ae4 env: fix comments about default values
fd6fe12b6 Fix scheduling when putting a device into looping
99b35ba79 Export the full lava-server version
da87efaa6 base-uboot.jinja2: add support for append_dtb and use_xip
babaef51d lava_scheduler_app: api: Add pipeline information to get_device_status
abe787872 Add a note on installing lava-dev
2b6ff1dca MASTER_CERT of lava-master should use secret key
d9e7e2c08 add recipients in notifications.yaml
44eb1b75b Move job outputs to sub-dirs based on submit-time
839b3ff19 Create directories with 0o755 by default
9459aae36 lava-master: call job.output_dir to get the path
8c9af4897 Fix description for devices and workers
541478930 Fix health-check tests by testing None and ''
68b33d572 Use job.output_dir whenever possible
d93df5f4f Add a management command to remove old jobs
bfd57121d Move unnecessary constants into base jinja template.
lava-dispatcher
===========
3be27df4 lxc protocol: simplify the tests
a5d9ebad Fix detection of missing ssh_host value in validate
234eef91 Fix calling of protocols after LXC change
477a36b7 tests: make ShellCommand.logger a DummyLogger
62867f49 Use images.validation.linaro.org files for unit tests
1868936c Silence logging in more unit tests
609c783c Fix download action name
b1701978 More tweaks to silence messages from the unit tests
29efa263 Adjust for pep8 checks in jessie
0733f89f Make sure the test_character_delay is used for all
commands in test shell
78d2509f Account for empty environment string.
ea09b19a Take namespace into account when counting test stages
734688c5 Declare namespace of the test suite in results
49a6b92c Drop noisy info log message.
4fd5d4af Drop unused imports and unused variables
678b216c Rework the removable action to allow sd cards
3b42122a Fix bug #2975 after qemu-nfs introduction
7462ea62 Expand LXC support to add devices from all sybsystems
b853865f Fix missing check for u-boot commands parameters
8293693c Replace invalid characters in test_case_id.
a94e0f24 Fix preseed/late_command appending
db03483a Fix Action names (use - instead of _)
9e836e64 Add a Command Action
bbd4ba25 Make the parser stricter about the block names
65b3a563 Allow to pass integers to run_command
b41326bc Raise JobError instead of NotImplementedError
073f3b34 update gitignore
00f5d33f log: limit the length of lines send other zmq
5b2b0746 LAVA-889 Fix handling of multiple test blocks
3bfdcdb6 Add support for use_xip
8df17dd7 Add support for append_dtb
3c110cae Remove unused imports
176a4d49 Add new utility function infrastructure_error_multi_paths
dc008eb5 Export the full version string.
daebfad1 Extend secondary connection fix to support primary
fae0ced5 Move unnecessary constants into base jinja template.
18f745bc device-types: add kirkwood-db-88f6282.conf
815e518d device-types: add at91sam9m10g45ek.conf
6ca0731f device-types: add at91rm9200ek.conf
003b53dc device-types: add at91-sama5d4_xplained.conf
9b2371cf device-types: add armada-3720-db.conf
5a767331 device-types: add armada-xp-gp.conf
8b050938 device-types: add kirkwood-openblocks_a7.conf
f5801d38 device-types: add alpine-v2-evp.conf
19e20276 device-types: add sama5d34ek.conf
49da704f device-types: add armada-385-db-ap.conf
ed7aa6b1 device-types: add armada-370-db.conf
9a104327 device-types: add at91-sama5d2_xplained.conf
ddb34cea device-types: add alpine-db.conf
f7642f58 device-types: add armada-375-db.conf
0d70c7dd device-types: add at91sam9x35ek.conf
a642d451 device-types: add armada-xp-db.conf
2a5ff9e4 device-types: add armada-7040-db.conf
1a439995 device-types: add orion5x-rd88f5182-nas.conf
ad855a62 device-types: add armada-xp-linksys-mamba.conf
58308ca1 device-types: add armada-388-gp.conf
6a2898bf device-types: add armada-370-rd.conf
cd1b4f47 device-types: add armada-398-db.conf
51925fb1 device-types: add sun8i-a83t-allwinner-h8homlet-v2.conf
4976d068 device-types: add sun8i-a33-sinlinx-sina33.conf
681480d5 device-types: add sama5d35ek.conf
db74d60e device-types: add sun5i-r8-chip.conf
40736b95 device-types: add sama5d36ek.conf
433be0a6 device-types: add imx6q-nitrogen6x.conf
ed4c82e6 device-types: add at91sam9x25ek.conf
ce7a488e device-types: add at91sam9261ek.conf
f87cdb49 device-types: add armada-xp-openblocks-ax3-4.conf
634442ca device-types: add armada-388-clearfog.conf
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
For 2017.5, we are planning on changing all of the action names
(self.name) to use hyphens to separate words instead of the current
mix of hyphens and underscores. This is to make future support more
consistent.
All V2 templates currently in upstream will pick up these changes as
part of the review process.
Any V2 templates which are not upstream and which reference timeouts
for specific actions may need adaptation, however, note that 2017.5
also moves a lot of the timeout settings to base.jinja2 anyway.
If you have test jobs which reference specific actions, these will
need changes too.
We're implementing a change in the dispatcher codebase which verifies
that underscore is not used in the self.name of any class inheriting
from Action, so all code reviews will also need updating.
https://review.linaro.org/#/c/19027/
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
This interim release includes a small set of fixes aimed at the LKFT
instance and HiKey support using LXC.
If your instance uses the pre_os_command support for fastboot with LXC
(not just pre_power_command), then note that this support now needs to
be called from within the LXC protocol.
See https://lkft.validation.linaro.org/scheduler/job/885/definition#defline77
for an example.
Changes:
lava-dispatcher (2017.4.post1-1) unstable; urgency=medium
.
* Production hotfix release for LKFT support.
3f99cc3 Add missing super call to WaitUSBDeviceAction
5436658 Fix indent typo
4847bf1 Put device rebooting info message at the correct place.
6b85382 Fix call to lava-lxc protocol for pre-os-command
bdb4d12 Fix test_lxc_api unit test
e9df320 LAVA-920 - Workaround ptable issue in firmware
7b4096c Allow LXC protocol to call pre-os-command
b33d04b Fix OE image support for HiKey.
d714a9b Add missing test shell helpers
9518e74 Add missing import
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
V2 healthcheck changes
======================
https://lists.linaro.org/pipermail/lava-announce/2017-March/000027.html
This support has now been delivered in 2017.4 using the lava-server
manage migrate-health-checks command. If used, the --clean option will
remove the old definition from the database for the device-type.
https://staging.validation.linaro.org/static/docs/v2/healthchecks.html#inde…
There is also a new migration status page to help admins track which
devices still need to be migrated to V2.
Branding message
================
The branding support now includes a message which can be added to the
home page for the instance. For example, a short message to describe
the main purpose of that instance.
Reminder about the LAVA repositories
====================================
Until Debian Stretch has been released and the new testing suite
(called buster) is open for new uploads, new production releases of
LAVA will only be available from the
images.validation.linaro.org/production-repo repository.
See https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…
See also: https://lists.linaro.org/pipermail/lava-announce/2016-December/000023.html
Changes
======
A longer list of changes this time as 2017.3 was skipped to allow for
the Linaro Connect BUD17 conference.
lava-server
===========
7e36443dd Restore UEFI boot method for HiKeys.
6679d548e Fix dragoboard410c boot sequence.
5082adc80 Move anchoring fix for navbar to base template.
187c8520a Allow to override U-boot tftp command
0d28de652 Handle OSError as well as IOError in metadata store
be04868ff Update migration status page for disabled healthchecks
5f689e930 Improve job page
ed4dee272 Expand the docs on arbitrary device_info elements
35a7378e0 Remove references to deprecated health_check_job
5e5e6890c Add a link to the migration page
84f88a4fc Allow to override U-boot usb command
043bfeea4 LAVA-916 - Restore adb connectivity with hikey in V2
4211de051 Don't force {BOOTX} macro in U-boot template
571d71313 Don't force dhcp in U-boot template
b0184585c Add a section on which file is what in /etc/
9701ad6a0 Adding timeouts to Juno template
b2b8d8851 Add content on developer workflow
e36889a81 Revert change for debug filter button
c06ed4a22 Fix bug 2927 - typo in OPTIONS for $MASTER_CERT
cfa0c02c5 Fix navbar problem with in-page anchors.
bd554a2c4 Add docs on boot commands.
624c79ee1 Fix bug #2925 - LDAP Configuration demo mismatch distro.py
1ddde696b Another parent timeout needed for qemu
c3f356c0b Allow boot_qemu_image action timeout to be overriden
b24fe09af Use standard paths to keep the unit tests running
7433bb4e4 Add note on installing developer builds
a0e7756c7 Add a migration status page
ce59699ad Tweak the device type docs
adf670e7d Update copyright year for the docs to 2017
b2428ad8c Small updates to the docs for test development
6610ce36f Allow to import token from another instance
d57e5217b Cleanups for advanced installation docs
0119fc57f Fix job submission page
0c4e68769 Dedupe the index page and add more entries
ec0aa40ad Improve device page
912c174b1 add notes on the new lava-tool features
a662f0ed6 Add a standard NFS test job for panda
e80ed1e13 Update docs on transfer_overlay
b0ae30b56 Update migration for job_status_trigger for backwards compatibility.
f28af7fbe Fix mustang doc examples and add uefi-menu
a755b4b12 LAVA-768 allow to disable hc for a device type
7a05b7941 Add support for a branding message
b18e8d293 log: add a link to the top and to the doc
0fd8c985e fix typo in rst header
75d411f14 Do not send a spurious POST submit request
d7f8d6322 Update docs for change in behaviour of log file UI
acd068173 LAVA-112: display status and role for sub jobs
bf144047f Check V2 devices have valid config before reservation.
0a2da3f92 Skip retired and obsolete device types
c106b17c0 LAVA-913 - Documentation for lxc persistence
14c733d30 Add another index entry for the example first job definition
b8629e0e3 Expand simple admin to include administration outline
43bb9da87 V2 jinja2 template for armada-8040-db
ca05a93d0 Add a command to migrate health checks to the fs
ad55286d6 doc: fix label
0b17a0496 Add qemu NFS docs
a8bf524f4 Update of doc examples
7bc903e70 LAVA-736: move health checks to the file system
1de7ea5b3 LAVA-912 - Document verbose for lxc protocol
643c23efc Remove deprecated (and unused) functions
91cff154b Add notes on when to add more workers
69df43a50 LAVA-904: Improve job page layout
3e3075f2f Reorganise the scheduler app test files
639600952 LAVA-910 allow iteration if vland tag check fails
1ebe9d217 Improve readability of testcase results
b7792c56e fix typo in index syntax
143b5f742 Allow boot_interface override
b85bc07a6 Update default interrupt_* in hi6220-hikey device type jinja2 template
a561326bc Add a test case for secondary connections
d6a441c16 Add Acer Chromebook 13 CB5-311
7ded576cb Add Rada Rock 2 Square
65e2b1e22 Add Renesas R-Car Starter Kit Pro
61edf0180 Add Samsung Chromebook 2
068005a9a Add i.mx6q sabre lite board
a84a509e4 Allow u-boot-interrupt timeouts to be overridden
b7a23dbfd Fix test job timeouts and fix doc
c94238613 Ensure extra_kernel_args is used from job context
77693f080 Clean up error handling in is_deprecated_json method.
8b5fa8130 Add a 'full docs' link to the front page
9ab49a8e9 Allow unit tests to run without kvm module
e47c5c005 Change job_status_trigger field type to a non-deprecated one.
6ec411a55 Tweak the authentication configuration docs
7ee493bb1 Tweak the authentication configuration docs
baa9f5ebb Update references for a permanent location.
da79c4be7 Add available architectures for qemu 'arch' validation.
65e4c35ec Prevent intermittent unit test failure.
7f9b8760d LAVA-215 QEMU NFS support (server side)
3f2f414e2 LAVA-619 document changes for bootz to zimage
8bbbd425f Ensure missing font is available for packaging.
0f2d8a4ff X-axis attribute in charts now use NamedTestAttribute table.
93cc20aa6 Fix export custom queries feature.
a8d6cab43 Display results sorted naturally by key in job logs.
e7aa614c5 Update the default length of the job result table.
e2c07c1ac Fix doc example YAML syntax to read from file
ce59d6b94 Fix sphinx build errors in manpage
63d5f6fa5 Add a check on all templates in source tree
d9eb3e1f3 Ensure character delays are supported by overdrive
75efcb00b Allow repeating test cases
93d620a53 Explain how connection-namespace parameter works.
9041b0dc0 Begin a section on common YAML syntax errors.
6521cc0f8 LAVA-894 - Document lava lxc device add / wait command
ee3e5964f Fix documentation - punctuations, links, bullets.
988d89a34 Add server side support for mustang EFI Grub.
525d9ed47 Fix code blocks in Boot Action documentation.
106f7f4fc LAVA-890 - Allow DUT to stay in Android OS
887175d82 Remove unused images
9f9e8d304 Fix unknown test case handling in tables
bff6ea8ff LAVA-888 - Download button in Device Dictionary page
3314966de LAVA-887 - Introduce fastboot sequence in device type templates
1a1be1ff2 LAVA-867 - Allow fastboot options in device dictionary
d876d1463 Remove legacy nexus deploy document from V2 docs.
2750804af Updating Juno jinja2 template to accommodate vemsd deploy action
2b6d3d277 Deprecate "pipeline-worker" in favor of "workers"
587bc8931 Add a "details" command to "device-type" and "workers"
96761e533 Add a command to manage devices
3b2f27ee0 Encode maintenance state machine in unit tests
730b57b81 Do not put into maintenance retired devices
c0adfd2c2 Add a command to go into maintenance
49abd8bf7 Fix state transition when OFFLINING a device
f337d7d1c Go from OFFLINING to RUNNING when a job exists
39b50ec15 Extend docs on simplistic testing
8e56315eb Workers: allow to set properties
944caa4fd pylint improvements
c42629220 Use continue in a loop, not return
61b6886f9 add notes on debugging multinode as admin
2ed1f35c7 Device tables: add more fields to select_related
d95ad2d5b job logs: Highlight all failing results
ace40b01a Generate less database queries
7fcbd7c68 Allow to use last version of Django Debug Toolbar
61069528d LAVA-862 - Integrate Google Pixel into LAVA V2
e7925e7a1 Update docs on publishing and attaching to tests
da99f08f0 Fix a typo in nexus5x device template used for unit test.
ba5a106db More stdout changes to clean up the test output
c84741460 Skip panda_lxc template check if lxc not installed
6a0a73d36 Fix a typo in documentation.
79ea7cedd Fix looping bug in template
466f1506a LAVA-841 - Document device_info attribute
d420b9d5f LAVA-856 - Integrate nexus5x into LAVA V2
de33008ec Autologin doc update.
lava-dispatcher
===============
7aac7ae5 Fix problem with self.host on secondary connections
530ca366 Ban / in test case names, replace with underscore.
31ee0012 If allow_fail is not set, ensure the command fails
e53a280f Remove old device_type config files which are unused.
80fb3997 LAVA-916 - Restore adb connectivity with hikey in V2
c57a102b Add debug output to the LXC device operations.
7deea0fb Add missing calls to super
341ac593 Writing connection info to namespace data
3ee2af87 Upgrade the boot commands output to info
a1a87daa Fix bug in checking return value of action run_command output.
604323b0 Add a unit test for primary ssh connections
561f78a6 LAVA-914 - Action run_command should return log on allow_fail
a8fb0d9d device-types: add Armada 8040 DB board
98fc1d4d LAVA-911 transfer overlay to the rootfs
e85af0a2 Report versions of critical software on the worker
bdc97bd5 LAVA-901 - Allow lxc debug output if requested in job
93aab7ab Add systemd packages as default for debian and ubuntu templates.
d01f01a3 Allow params for install git repos support
eb821dc8 testdefs: rename the extra results keys
f262b21c testdefs: fix TOCTOU issues with test definitions
07affb55 Print warnings in validate and not in populate
3de0aac9 Make empty revision message more friendly
685eea49 Do not populate append when using tmpfs GuestFS
22a7ba93 Add missing imports
a1baf85e Add missing JobError import.
9e8c3c2f Improve validate and job summary
b1100080 Catch error in secondary connections
7428a9ae Improve git error logs
949b3fcd Rework the timeout settings in job and device conf
6933ac63 download: improve error message
304c39f7 LAVA-852 - LXC fastboot support for persistent devices in V2
fc28aafe Add an error type to the job result
522946b8 Make '/sys/class/misc/kvm' a constant.
f9a1b381 tests: use the DummyLogger whenever needed
ee11c796 tests: use permanent urls
538ff35f Context architecture validation for qemu.
bd88adab Check for enabled kvm module on workers.
28c66090 Fix bug #2898 - Job doesn't exit on fastboot-deploy failure
2879c019 Handle qcow2 conversion errors
7daf65d1 Action: use LAVAError base exception class
603fd049 shell: remove unused exception handling
da97077d Add missing import and remove uneeded one
beccc4da LAVA-215 QEMU NFS support (dispatcher)
c2e3ee34 Use DummyLogger for grub and add .target()
ed3519d2 Get uboot header length from utils.constants
7df2904d If no kernel type given in deploy, use the boot type until deprecated
82237860 Ensure the deployment data shell is exported.
108ef4a3 Add the commit-id to the lava testdef results
1e3dd3ce run: improve error handling
41b08257 lava-dispatch: use the new LAVAError base class
171e8b65 device-types: add Allwinner A23 EVB board
5898747b device-types: add OrangePi PC board
4ffb4c1e Simplify lava-test-shell by removing stdout.log
a5222296 validate: improve error handling
c2df0b20 Remove the message from cleanup()
78c334c2 Simplify test/shell.py check_patterns signals.
68f1c822 test-runner: remove redundant functions and files
3fe2105e Fix bug #2888 - adb root command from job breaks connection to device
22157cd2 Add a base class for LAVA exceptions
44af91c3 Support using UEFI Menu to load Grub using PXE
b667f2fd Use LAVA exceptions
1690ed95 Pylint updates for deploy actions
e538b5d8 LAVA-890 - Allow DUT to stay in Android OS
bbcf76fd Raise a TestError when the test installer fails
0d45d8fa test-runner: use the current shell
27593024 Blacklist brltty on workers
e2593966 Allow reusing of serial connection from another namespace.
cfe9821b LAVA-867 - Allow fastboot options in device dictionary
d5094f28 Fix unknown usage command in lava-test-reference
381eca8d LAVA-881 - Sequence list for fastboot pipeline
e84c00b1 action: be consitent when printing result duration
82d7f82f Print the result of a test run only when avaiable
1d09ece0 Catch python3 errors to clean up output
833d7207 python3: only show requests logs above warnings
5fb1e8a0 rpcinfo: also capture the output
496c7181 Sanitise default mkimage_arch and kernel handling
a6838ecb LAVA-862 - Integrate Google Pixel into LAVA V2
d2f968e3 Follow up to stabilize dragonboard 410c.
8ab18e05 rpcinfo check needs to specifically check NFS service
61777bb0 Refactor wait usb device action to stabilize dragonboard 410c.
e7649e64 Remove references to device_path in device configuration.
3fd11d81 LAVA-856 - Integrate nexus5x into LAVA V2
d970298d Adding new action for monitoring USB mass storage devices via udev
94876fd7 pyocd: allow multiple images to be flashed
26528b0c Detect bootloader resets.
09650ab1 Adding a Versatile Express Firmware deployment action
605fe529 Detect errors during auto login.
00306b79 Job: log the job result as a logger.results
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
2017.4 will deliver support for migrating V2 health checks out of the
database and into /etc/lava-server/dispatcher-config/health-checks/ on
the master. This has several benefits for admins.
* Health checks from 2017.4 onwards will be based on the device
configuration, not the device type. This applies to V1 devices and V2
devices.
* Multiple instances can all have the same V2 health checks controlled
in VCS using configuration management tools.
* V2 devices of the same device-type can have different V2 health
checks according to the Jinja2 template specified in the extends field
of the device dictionary. Submissions to such devices would need to be
managed through device tags.
* Devices which are V1 only (is_pipeline is False in the django admin
interface) will continue to use V1 health checks from the database,
based on the device-type.
* Devices which support V2 (whether these devices are exclusive to V2
or not) will only use V2 health checks from the filesystem, based on
the device dictionary of each device.
This does mean that devices which are currently both V1 and V2 will
need to have a V2 health check from 2017.4 onwards or the device will
no longer run a health check at all. Admins may choose to submit and
monitor regular V1 testjobs until all test jobs for those devices are
V2 but those test jobs will not be able to take the device offline
automatically.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
There is a bug in python-lockfile available from jessie-backports:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=858835
This affects fresh installs of LAVA when LAVA is installed directly
from jessie-backports without first being installed in jessie.
The underlying problem is with apt which pulls in the updated version
of packages which have a backport available even if the requested
package does not explicitly ask for the backport of that package.
The problem shows up with lava-coordinator first - the package fails
to install cleanly because the daemon cannot be started correctly.
To fix this problem, simply instruct apt to install python-lockfile
from jessie, not from jessie-backports:
apt install python-lockfile=1:0.8-2
lava-coordinator will then install correctly.
This change will persist over later upgrades as apt does not consider
jessie-backports as upgrades of jessie - except in the case where the
package is not yet installed.
Other daemons also now need to be restarted:
$ sudo service lava-server restart
$ sudo service lava-master restart
$ sudo service lava-slave restart
Future builds for the staging-repo and production-repo will force only
version 1:0.8-2 of python-lockfile as these builds are done on and for
jessie.
Systems already running Stretch or unstable do not seem to be affected
and can safely use python-lockfile at version 1:0.12.2-2 - this is a
particular problem inside jessie-backports. However, this does mean
that until this bug is fixed, users running Stretch or unstable should
build their own developer packages instead of using the LAVA
repositories on images.validation.linaro.org. We currently do not have
capacity to build separate packages for jessie and stretch.
(LAVA does not recommend using Stretch or unstable for production
systems until Stretch has been released and stretch-backports becomes
available.)
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Reminder about the LAVA repositories
====================================
Until Debian Stretch has been released and the new testing suite
(called buster) is open for new uploads, new production releases of
LAVA will only be available from the
images.validation.linaro.org/production-repo repository.
See https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…
See also: https://lists.linaro.org/pipermail/lava-announce/2016-December/000023.html
2017.2 release
==============
More updates to the jinja2 templates in this release and some new templates.
diffstat output:
lava_scheduler_app/tests/device-types/apq8016-sbc-uboot.jinja2 | 3
lava_scheduler_app/tests/device-types/arduino101.jinja2 | 5
lava_scheduler_app/tests/device-types/b2260.jinja2 | 8 -
lava_scheduler_app/tests/device-types/base-uboot.jinja2 | 26 ++--
lava_scheduler_app/tests/device-types/base.jinja2 | 11 +-
lava_scheduler_app/tests/device-types/d02.jinja2 | 4
lava_scheduler_app/tests/device-types/d03.jinja2 | 1
lava_scheduler_app/tests/device-types/frdm-k64f.jinja2 | 30 +++++
lava_scheduler_app/tests/device-types/hi6220-hikey.jinja2 | 7 -
lava_scheduler_app/tests/device-types/highbank.jinja2 | 2
lava_scheduler_app/tests/device-types/juno-uboot.jinja2 | 1
lava_scheduler_app/tests/device-types/mustang-uefi.jinja2 | 3
lava_scheduler_app/tests/device-types/nexus10.jinja2 | 1
lava_scheduler_app/tests/device-types/nexus4.jinja2 | 1
lava_scheduler_app/tests/device-types/nexus9.jinja2 | 1
lava_scheduler_app/tests/device-types/nrf52-nitrogen.jinja2 | 9 +
lava_scheduler_app/tests/device-types/nucleo-l476rg.jinja2 | 23 ++++
lava_scheduler_app/tests/device-types/nxp-k64f.jinja2 | 1
lava_scheduler_app/tests/device-types/nxp-ls2088.jinja2 | 12 ++
lava_scheduler_app/tests/device-types/overdrive.jinja2 | 1
lava_scheduler_app/tests/device-types/r8a7791-porter.jinja2 | 11 ++
lava_scheduler_app/tests/device-types/stm32-carbon.jinja2 | 11 +-
lava_scheduler_app/tests/device-types/x15.jinja2 | 55
++++++++++
lava_scheduler_app/tests/device-types/x86.jinja2 | 1
24 files changed, 178 insertions(+), 50 deletions(-)
Changes in the 2017.2 release are described below, including the
short git commit hash, include:
Use the git commit hash to go directly to that commit using a URL like
https://git.linaro.org/lava/lava-server.git/commit/?id=266b2e633
Commit messages including a reference to LAVA-NNN relate to projects.linaro.org
append the reference to https://projects.linaro.org/browse/
e.g. https://projects.linaro.org/browse/LAVA-854
lava-server - https://git.linaro.org/lava/lava-server.git/
-----------
9df8b354b Remove whitespace from case names
9354cd585 Uboot parameters fixes
297f27900 Fix documentation for auto_login
64b549e9a Fix group visibility and hidden device types
b211434de Document the use of a list for bug 2870
eb8a7bec3 LAVA-854 - Device type template for x15
ad81f33fb Add device template for renesas r8a7791-porter board
8c4d44ae4 Add device template for nucleo-l476rg
112ec6487 Remove send_char, and fix some broken character_delay settings
8f615cd6b Update reviewers handling
0fe482ffb b2260: remove duplicate ip_args from extra_kernel_args
a72c86732 LAVA-847 protect metadata store IOError
53a3c778e If a text_offset is used, quote it to ensure it is a string.
51351e18c Add documentation of the dispatcher-config support
7a1bfb649 Tidy up the Device-type templates for IoT devices
0b5de4dcb Allow lava_scheduler_app to unittest logs to stdout
bc807def4 Fix sizelimit warning handling for V2
13b691615 Summarise job timeouts with example and notes
b5c2ea1a7 Improve notes on portability.
65ae7d644 Fix internal server error due to unpacking TestJobUser.
83e8fa075 Migration doc improvements
9eb145050 Add a unit test for the ethaddr support in b2260
91eb07a36 Creating and restoring backups for V2
479a6295f Add support for setting ethaddr
52b5cf06d Add triage documentation for power failures
526a6e298 LAVA-840 - Remove documentation for USB_DEVICE_WAIT
3415d5319 Allow .py files in examples/sources to be packaged
be33389bf Allow notifications for jobs in Running status.
4d6afd607 Fix blacklisted bug in notifications.
58b48a8f0 Catch all errors during resubmission.
40b010238 Tag admin clean support must return name
14b15f40b LAVA-734 compatibility docs
dd18c26e1 scheduler: do not cascade deletion for some objects
5415eafa7 Do not create unused TestJobUser
2f3cd8389 Bug 2796. Add submitter username to job_details API.
08cd6c01f Revert "master: set the master identity"
be28c8c7b LAVA-814 - Explore pyudev for usb device wait
9150c9973 LAVA-832: Rework device-types management command
70e81493e LAVA-762 link level in timing to pipeline definition
c276df83f LAVA-728 extend ZMQ example to submit and wait
f6a0456a1 LAVA-731 document remote worker issues
41c67ae3b Add device template for NXP-ls2088
ee8964c7b Remove spurious call to save()
2a79a622d Ensure base_ip_args can be overridden
affa68b23 Add a command to add and list workers
10e6ea970 tokens: allow format as CSV
b0b5a1d30 Remove unused config value LAVA_CONFIG_VERSION
69138ac89 Add a command to manage tokens
e09beb1a4 Remove unused custom commands
905a0f9be Fix multinode job definition page
bb0fdc7a8 job logs: Highlight failing results
e1a125aa8 master: set the master identity
b7cb28e49 Fix device schema and test for arduino template
d411416bf LAVA-707 update result case and metadata views
15fc5c62f LAVA-140 - update running job calculation
9fcf92cfd LAVA-515 restore line numbers to YAML definitions
b7266712b LAVA-824 skip unused device types in running table
de6a141aa Expand docs on namespace
c54a5e262 events: fix documentation
lava-dispatcher - https://git.linaro.org/lava/lava-dispatcher.git/
---------------
ba779908 testdef: set "set +x" to avoid duplicating SIGNALS
ae2ab424 Enforce a download order to produce predictable pipelines
55a39b92 Remove unnecessary boot action in fastboot x15
81b178fc Add a unit test for db410c pipeline generation.
aa96dd75 LAVA-839 - Integrate X15 into LAVA V2
14acd2e2 Fixup u-boot prompts sanity
829fa380 Support CMSIS-DAP devices that do not reset after being flashed
d09a7a74 Simplify send_char support
031ba8e8 Ensure dispatcher.yaml is available for admins
aa6142dd Detect pdu controlled device using availability of power command.
c54891b9 Properly isolate stdout and stderr in unit tests
2d8b3110 Ensure lava-test-reference is available for release
555b6be0 Skip one test if lxc-info is not installed
fc42bd89 Fix support for persistent NFS
57c96b4b Remove duplicate prompt handling.
b0d39204 LAVA-844 allow string and dict for install: git-repos
76998b75 LAVA-830 Use the configured TFTP directory
1f01283f Partially revert a1658c by using a DEALER socket
9045292b Allow boot commands in the job definition
af34dfcb Extend the timeout unit tests to boot actions
9135c5aa Add support for CMSIS-DAP IoT devices and cleanup PyOCD
96dcc7d7 Only match udev events when device is added
a1658c7e slave: use a ROUTER socket to connect to master
35937826 Fix prospector warnings
e4e2fb97 etc/lava-slave: fix documentation of WARN log level
b45cb005 Remove unsed imports
93a282bf Job: log known error status when leaving
ee404039 lava-dispatch: allow to run without zmq logging
26daca49 lava-dispatch: work around a variable clash
c62e1ee9 LAVA-814 - Explore pyudev for usb device wait
989cd97b LAVA-827 ensure the case name is a valid URL
570c1b1e Do not reset the timeout before each connection
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Reminder about the LAVA repositories
===========================
Until Debian Stretch has been released and the new testing suite
(called buster) is open for new uploads, new production releases of
LAVA will only be available from the
images.validation.linaro.org/production-repo repository.
See https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…
See also: https://lists.linaro.org/pipermail/lava-announce/2016-December/000023.html
2017.1 release
===========
This release includes substantial updates to the jinja2 templates. If
you have made local changes to these files, you will be prompted
during installation.
/etc/lava-server/dispatcher-config/device-types/apq8016-sbc-uboot.jinja2
/etc/lava-server/dispatcher-config/device-types/arndale.jinja2
/etc/lava-server/dispatcher-config/device-types/b2120h410.jinja2
/etc/lava-server/dispatcher-config/device-types/base-uboot.jinja2
/etc/lava-server/dispatcher-config/device-types/base.jinja2
/etc/lava-server/dispatcher-config/device-types/beaglebone-black.jinja2
/etc/lava-server/dispatcher-config/device-types/cubietruck.jinja2
/etc/lava-server/dispatcher-config/device-types/highbank.jinja2
/etc/lava-server/dispatcher-config/device-types/juno-uboot.jinja2
/etc/lava-server/dispatcher-config/device-types/mustang.jinja2
/etc/lava-server/dispatcher-config/device-types/panda.jinja2
/etc/lava-server/dispatcher-config/device-types/qemu.jinja2
Changes in the 2017.1 release are described below, including the
short git commit hash, include:
Use the git commit hash to go directly to that commit using a URL like
https://git.linaro.org/lava/lava-server.git/commit/?id=266b2e633
lava-server - https://git.linaro.org/lava/lava-server.git/
--------------
266b2e633 LAVA-795 - Expose lava-publisher event socket settings.
f4b76e48e Fix ordereddict failure
9d3ae86cc Use dpkg-query which is available on all systems
080c3184f Update the device-type context help page
46a055a3b Fixup some jinja2 templates
cecefbfef Timing: improve the description
5bef1d877 LAVA-821 - add support for lava-test-reference
3b26fb234 LAVA-820 record lava-server version in job metadata
ff71a42a6 Append to existing metadata store results
0c9abf643 Fix RST whitespace typo.
05b96af3c Merge timing page into the job page
edc4431f1 Restore access to chart Action buttons
69709881e Improve the timing page and port to the new log format
a00c02aa8 LAVA-535 Make devicetags case-insensitive
36617b955 LAVA-743 - handling complex test operations
50084f1b5 Omit summary button for V2 definitions.
5c8bf4643 Upgrade the dispatcher v2 parser interface.
22409d543 master: send the dispatcher config to the slave
lava-dispatcher - https://git.linaro.org/lava/lava-dispatcher.git/
----------------------
64ae9131 Report the bootloader load addresses used
2085ffb8 Skip parsing of kernel messages for ssh connections
00fa2d24 Improve allow_fail fault handling
dce680f3 Simplify the connection session handling
ea18594a Use dpkg-query which is available on all systems
2cc64b70 Reinstate git version_tag in version.py
5c05bdae Improve logging
50236943 LAVA-821 - lava-test-reference support
a118e943 Add missing wait definition in base class
2dc955f6 Allow timeout during Linux Kernel message parsing
0404a419 LAVA-819 - record lava-dispatcher version in logs
542effdf Raise an Infra error when download fail in run()
29a232c1 Fix Download retries crashes
12e5789f monitor: make test a bit faster
a90a5184 Fix computation of action duration
ffd1d176 Fix retry logic (partial revert of 7bbd706d)
36ca3c21 LAVA-818 - errors should set the job as Incomplete
f5ac0328 Skip generation of invalid test_case_id
9e5e56c6 Re-enable support for nose tests
bb89772f Merge and re-use BootloaderCommandOverlay and
BootloaderCommandAction
f3864c2f job: fix cleanup when validation failed
8dd450c5 run_actions: clean error messages and exceptions
ec44db64 Properly compute the timeout
f3058a12 atexit is not needed anymore for cleanup
7bbd706d Improve error handling and reporting
a5e0e678 Fix issue when downloading two files with the same name
0d3773ea Call finalize directly in job.cleanup
af608eb2 Remove unused setter, properties and __call__
f946a21d validate: do not extend action.errors
4504aeb9 slave: use the dispatcher configuration
925019c8 Use the dispatcher config to get the dispatcher ip
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Enable the LAVA repository
====================
2016.12 will be the last chance to update of lava-server and
lava-dispatcher into Debian Stretch before the release as Debian 9.0.
This means that users of jessie-backports should now enable the LAVA
production repository - see
https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…https://lists.linaro.org/pipermail/lava-announce/2016-November/000019.html
Until Debian makes the 9.0 release, jessie-backports cannot receive
updates of the LAVA releases in 2017, so all updates will only be
available from the LAVA repositories. Shortly after the release, a new
stretch-backports suite will become available and jessie-backports
will be opened again.
In accordance with the Debian freeze policy, it will take 10 days for
the new uploads to migrate into stretch, not the previous 5, so an
upload to jessie-backports will be made at that time.
Note: this may be an issue for armhf or arm64 workers - if you are
using such workers, let us know so that we can prioritise the changes
required to build lava-dispatcher for those architectures.
Updates from jessie-backports
======================
A few updates are required from jessie-backports with 2016.12: lxc and
python-pyudev
The lxc update ensures that jessie and stretch containers can be
correctly created on jessie.
The pyudev support enables lava-dispatcher to monitor udev activity
for certain operations.
These updates can be applied before installing 2016.12, if desired:
$ sudo apt install -t jessie-backports lxc python-pyudev
lava-tool
======
A new version of lava-tool is also available, including a new way of
handling the authentication tokens and the ability to use shortcuts
(like "local" instead of "http://localhost/RPC2").
https://tracker.debian.org/pkg/lava-toolhttps://tracker.debian.org/news/825902
The new version of lava-tool is available via the LAVA staging and
production repositories and the package available in jessie-backports
will be updated in due course.
Unfortunately, it proved not to be possible to export the existing
authentication secrets into the new backend or to support both
backends simultaneously. All users will need to re-add all their
authentication tokens to use lava-tool after this upgrade. The change
has several benefits, including the ability to list and remove
authentication configurations using the new backend and the removal of
a whole host of problems related to using the old backend in
environments where lava-tool was not running in a full terminal. In
addition, this new backend allows lava-tool to support shortcuts which
prevent problems with repetitively entering long and complex URLs.
On installing the new upgrade, users will see the NEWS message:
https://github.com/Linaro/pkg-lava-tool/blob/master/debian/NEWS
lava-tool has changed the backend for token storage.
This means that existing authentications will not be usable to this
new version. This change has been made to fix persistent problems with
the python-keyring support, including:
* DBus errors when used over SSH
* Interaction with gnome-keyring causing authentication failure
* Inability to list or remove authentications added to the keyring
* Inability to work with other keyring solutions.
The new backend is able to list and remove authentications. The new
support also removes the need for a default password on the user
keyring, so lava-tool will no longer pause waiting for password
entry.
2016.12 release
============
Changes in the 2016.12 release are described below, including the
short git commit hash, include:
Use the git commit hash to go directly to that commit using a URL like
https://git.linaro.org/lava/lava-server.git/commit/?id=b19b9648a
lava-server - https://git.linaro.org/lava/lava-server.git/
-----------
541ba7b0a Enable ssh secondary connections on juno-uboot
2d6bb2fcc Handle errors which cause invalid description data
b19b9648a Fix template_mismatch check to use extends
d7f71eae3 Update ipxe x86 nfs to use base_ip_args
fa6b8b3ab Add a note on release-notes on lava-announce
d007d1819 Fix a crash when a device does not have a device dict
d46d6e4fd Protect against invalid parameter submissions
431fd2fc0 Allow pipeline jobs to be used with XML-RPC job_output
f3bf855ba Add Copy to clipboard functionality to MultiNode
19d814e0c Fix some failing tests due to new namespace changes
86950e985 LAVA-745 - expose device_path for all types
083d67d54 Add device-type template for arduino101
fd99c576e Fix UI submission error.
708a38a23 Add SSH support for overdrive
e838de3ba Update Linaro lab links to actual site
d8e127b74 Allow flash_cmds_order to be set
92eefe4aa Adding extra boot commands for secondary media on Juno
74310353c LAVA-792 - add shorthash to changelog
8f860a1e9 Keep result blocks on a single line with MultiNode
3b1d6654e LAVA-499 - Device type template for dragonboard-410c
5c46ea4ab Provide a default fastboot flash commands order.
a61ab2501 Download results XMLRPC support.
347fb3fd3 Allow to forward the messages to additional sockets
4d0753c3a Allow validate_pipeline devices to filter by device type name.
a859e5ede publisher: improve logging and fix logfile owner
f0002efb1 Fix sphinx indentation error
9f9bd71b9 Correctly expose TestSet data in results
c6f9b1df8 Handle decimal conversion error in test case
1769d8b6b Expand hints on new device-types
eec1ca1d0 master: expect a protocol version with HELLO messages
cd8d109f7 master: add the job id to the log messages
a0ac2d66a Add absolute_url to job_details API.
951ab9f68 master: device configuration is empty for secondary media
c0575503d master: improve error reporting when reading env file
f551daa49 Add notes on unattended upgrades
1d2e25d1d Allow override of nfsroot options, not arguments
edaad5419 Update docs to reflect reality of from: support
09503a482 Manually revert d31a8462: errors are already saved
3516b856d add some notes on packages needed for LXC usage
b5ee513bb Rename 'usb-showup' to 'usb-device-wait'
406832582 Drop misleading hostname from device metadata
6a6ee6f4d Handle change to git.linaro.org frontend
d85f99e1b Fix crash when description.yaml is empty
37a94aafc Fix prospector warnings
080499d18 Remove references to wsgi
267d22302 Paste job definition to clipboard as text and not as HTML.
ef09a9698 add a link to the device tag glossary entry
08bd011a4 Update aliases migration for review change
7010775b6 Move metadata extra content to files
95ed9a3ba Expand notes on debugging test jobs and test cases
6d772bb0d Update for pycodestyle pep8 failures
dd58a1182 LAVA-798 - lookup device-types by alias
104feb593 Enable LXC for existing device-types.
376a80021 LAVA-797 - Document changing USB_SHOW_UP_TIMEOUT in job
3e8dbfeca all_devices() api call to do one big query instead of many small ones.
d27f7a4d5 Fix for Debian reproducible builds.
ac8bb8e2a Fix bug #2596 - No result collected for multinode jobs
7b3adad0a All jobs will have a definition, so show it by default.
1e6992f6b Set boot_character_delay in x86 template
5cc0a4746 Extend x86 unit test to check character_delay support
ad4f8780f Combine interfaces block for juno and vland
bb2ecf716 Allow changing the timeout for u-boot-interrupt
19de59978 Add stm32-carbon jinja2
lava-dispatcher - https://git.linaro.org/lava/lava-dispatcher.git/
---------------
19fd79a9 Update preseed support for namespace changes
ef352abf Update zephyr sample jobs to use LAVA functional test urls
1301258a LAVA-498 - Support dragonboard-410c with lxc
1e0b60d2 Ensure pyudev version is >= 0.21
7d6a8134 Update the DFU boot code for namespace changes
161e8b47 Ensure installer tests share the final connection
567ddf0d remove duplicate call
e49bd971 Fix namespace error in flash_cmds_order retrieval.
cb04549d Set connection for secondary media deployments
dcd75e7b Use correct suffix for secondary media deployments
2a7b386f Declare the original ramdisk as compressed
fd73e937 Support DFU devices
6aa7280c LAVA-504 - Dynamic fastboot partition support in pipeline
9228901a LAVA-809 - Show HiKey kernel boot messages
774c7557 Extend namespace fixes for secondary connections
dc5bec22 Action: Always call super() in validate()
9b983cdf GitRepoAction: do not call validate twice
bd112aab Make namespaces common to all dynamic data operations
2e83e2ca Prevent invalid measurements being reported
1768c181 Fix testlevels for multi-test LXC jobs
0a49da8e Give namespaces to all actions and support simpler code flow
2423a123 slave: send the protocol version with HELLO message
983d3775 Ensure fixupdict is checked in test_case_result
08f79918 slave: writing an empty string to a file is valid
c6f382ef Move cleanup_actions from Action to Job
bc56ca24 Remove unused Action, Pipeline and Job members
1256f31e Run.sh should use set -x
a726be8f Allow install-steps to use the git-repo directly
b0ca553a Create a useful exception message
077aca26 slave: don't crash when lava-dispatch can't start
543ed4af Honor USB device wait timeout and add device before returning
connection
0e07089f Fix problem with duplicate test shells failing
f14747c1 Use dnf instead of yum on Fedora version greater than or equal to 22.
efdaabc1 More fix for usage of LXC with a device.
2d188baa Limit repetitive kernel message log entries
c3455a11 Allow newline line separator to be overriden per bootloader
01a482f6 Fix for Debian reproducible builds.
62c07465 Indicate that character delay is in milliseconds
c1f79680 Remove hardcoded character delay in ipxe
e43552b3 Fixing secondary media deployment
e885c881 device-types: add Renesas r8a7795 Salvator-X board
1374289b device-types: add Amlogic meson-gxbb-odroidc2
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Everyone using LAVA should be aware of V1 and V2 and that V1 is going
to be turned off. The plan for how to do that has been discussed
within the team and within Linaro. The most recent presentation on the
migration can be seen here:
http://connect.linaro.org/resource/las16/las16-503/
(Note: the video will auto-start, be aware.)
The slides are available on slideshare:
http://www.slideshare.net/linaroorg/las16503-lava-v2-migration
The dates for each step in the migration are not finalised but will
occur during 2017, this is advance notice.
The highlights of the migration are:
0: During 2017, regular announcements will be made on this list before
each major change. In addition, the regular release announcements will
include details of the major changes in that release.
1: The first major change will be that a new release of lava-server
and lava-dispatcher will *disable* V1 submissions entirely. Any queued
V1 test jobs will become invalid upon this upgrade,
2: A later release will remove V1 documentation and start removing the
V1 source code.
3: A release will be made which *forcibly deletes V1 test jobs,
bundles, filters and image reports*. This is required to complete the
removal of the V1 source code.
4: The final step of the migration is a release containing no V1
source code at all. Only V2 support will remain. This is essential as
the V1 source code is currently blocking some useful additions to V2
as well as plans for future development. The data cannot remain
accessible without the V1 source code. There is not and can not be any
support for converting V1 data to V2.
Please take this chance to read up on the V2 documentation, including
how Results, Queries and Charts replace the deprecated V1 support from
filters and image reports.
Work has started upstream on making sure the migration runs smoothly,
especially when deleting large numbers of V1 test cases.
Remember: Once the first major step is released, upgrades will stop
running any V1 test jobs. Subsequent releases will include changes
which force the deletion of your V1 test data.
If you want to preserve your V1 data, consider setting up a read-only
reference instance which is then pinned at a particular release of
LAVA to prevent deletion of the V1 data.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
There is a known bug with QEMU 2.7 which is affecting installations of
LAVA using QEMU from jessie-backports.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=840853
The problem manifests in the apply-overlay-guest action because QEMU
is unable to process the GuestFS operations due to a missing file
which is in the seabios 1.9 package but not in seabios 1.7 - so V2
QEMU test jobs hang at this point.
If you are using QEMU from jessie (version 1:2.1+dfsg-12+deb8u6) then
there is no problem and nothing needs to be done.
If you are using QEMU from jessie-backports (version
1:2.7+dfsg-3~bpo8+2 currently) then there are two simple remedies.
0: You can find the updated seabios package in the LAVA production
repository. See
https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…
If you have the production repository already available, you can
simply apt update and apt upgrade to install the new seabios package.
1: If you are not using aarch64 QEMU tests or the QEMU UEFI support
for LuvOS, then the version of qemu in jessie will work just fine. To
downgrade, specify the version string to apt:
$ sudo apt install qemu-system-x86=1:2.1+dfsg-12+deb8u6
qemu-system-common=1:2.1+dfsg-12+deb8u6
It is also worth downgrading other qemu packages at the same time.
Check your current install with:
$ dpkg -l | grep qemu
The relevant bug has been re-opened and a backport of seabios has been
uploaded. An updated backport of qemu has also been requested which
would require the installation of the backported seabios 1.9. So if
this bug is not troubling you immediately, you can simply wait for
seabios 1.9 to be available in jessie-backports and install it.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
2016.11 includes a move away from libapache2-mod-uwsgi and
libapache2-mod-wsgi to gunicorn support. This has been done to make it
possible for lava-server and lava-dispatcher to migrate into Stretch
ahead of the Debian 9 Stretch release. (see
https://lists.linaro.org/pipermail/lava-announce/2016-November/000019.html).
This should allow us to upload 2016.11 to jessie-backports in due
course.
See: https://lists.linaro.org/pipermail/lava-announce/2016-October/000017.html
for details of the changes for gunicorn support. In particular, if you
are using apache, before installing this upgrade, ensure that these
commands have been run on the master:
$ sudo a2enmod proxy
$ sudo a2enmod proxy_http
$ sudo service apache2 restart
These commands are safe to run before upgrading to 2016.11 but will
need to be run before 2016.11 will actually run.
This release includes all changes in master since the 2016.9 release.
The release has been uploaded to Debian unstable, it should appear on
the official mirrors shortly. It has also been uploaded to the
production repo.
We are anticipating that there will be an interim release ahead of
2016.12 for additional LXC support, more details will follow via this
list.
lava-server (2016.11-1) unstable; urgency=medium
.
* New production release aimed at Stretch.
Release details available at:
https://projects.linaro.org/browse/LAVA-789?filter=12078
* Drop dependency on libapache2-mod-uwsgi and libapache2-mod-wsgi.
Replaced by dependency on gunicorn.
* Require sphinx 1.4 or later for correct theme support.
* Downgrade linaro-image-tools and fuse to Recommends
* Update compat version.
* Highlights of upstream changes:
Note about removing adb and fastboot packages in LXC admin doc.
LAVA-789 - Document LXC support in V2
LAVA-788 - link to sub_jobs_list from results
Expand user notification documentation.
Move jinja template tests to unittest
Adding target interface entries to juno-uboot.jinja2
Add docs on notifications
Update home page documentation links
Improve documentation on metadata and job_name
Fix up glossary page
Tweak docs handling standard test jobs
Example jobs need auto_login support
Fix missing create_device_database ref
Port the documentation change for essential roles
Tweak the doc build options
Add notes on multiple hacking sessions
Update debugging docs for tmp file changes
LAVA-65 document recording measurements
LAVA-780 stop referring to wheezy images
Avoid crash in master on early failure
Remove beautify.js and fix yaml URL submission.
Update screenshot for cancel button fix
Fix visibility of the Cancel and Admin buttons
Update docs to reinstate web UI submission
Add a helper to add devices from cmd line
Remove old link about LAVA packaging which contains stale information.
LAVA-749 - Add actions column to results pages.
Ensure lava-server-gunicorn restarts cleanly
results: do not crash if the description is empty
Job submission UI for v2.
Add copy to clipboard button for job definition.
Turn all action durations into result measurements
Updating juno-uboot.jinja2
Add a log file for gunicorn logs.
Fix the LAVA logo
Ensure the V2 layout is included on jessie.
bootstrap: do not use the minified CSS
Results: fix HTML syntax
Use sensible default for interrupt string in HiKey jinja.
LAVA-749 - Improve query pages
Fix unit test to have unique test definition names
Add pre_* commands to base.jinja2 that will be applied conditionally.
Remove unused dependendies on lava_dispatcher
Fix a variable redefinition in list comprehension
Update contents and organise toctree
Allow changing the timeouts for auto-login-action
Add documentation on QEMU standard kernel tests
Remove old mention of JSON
master: set the failure_comment
Add a template for highbank using base-uboot
Remove hardcoded values from base
Fix profile page error due to wrong device health history url reference.
metadata: fix a crash if the description is invalid
Cover lava-coordinator in changes for V1 workers
LAVA-522 Link in existing docs on replacing vmgroups
Extend the qemu options docs
Fix index links and debugging doc links
LAVA-735 describe anonymous access to JIRA
qemu.jinja2: use qemu-system-i386 for i386
Extend docs for disabling V1 on a master.
Fix HTTP 500 on query views when a group is used.
LAVA-765 V1 MultiNode sub_id correction
Fix jinja2 syntax issues in base-uboot
Fix sub_id assignment in V2
Improved debugging docs
Add content for growing your lab.
Improve first device and first job pages
Update apache docs for gunicorn
Update hacking sessions and lava-network docs
Enable vland for overdrive
Fix uWSGI configuration file
Add notes on removing V1 from dispatchers
Add a helper to add device types from cmd line
DeviceType admin: fix has_health_check computation
Replace ip=dhcp with {{ base_ip_args }}
timing: add the corresponding timeout
Fix typo in the title of the result download link
Add ip=dhcp for juno with a base default
Fix page to show health history of my devices and not my device type.
Fix error on 'Devices Health History' page in django 1.10
Fix user profile page error in django 1.10
Create a base uboot jinja template
Use Gunicorn instead of uWSGI
wsgi: use the Django public API
index: add the missing title
Mark support for LAVA in Debian testing.
Expand the context schema for extra_options
Add the new LAVA logo to the front page of the V2 docs
Warn if a pipeline device does not have a template
master: don't generate the description
Add nxp-k64f jinja2
Add nrf52-nitrogen jinja2
Add drafts of new logos with SVG.
Extend base timeouts and convert to minutes
Handle inline definitions using only install steps
job details: removed unused template variable
master: don't print too many logs in debug
Fix build errors and simplify index pages
Add support for overdrive device type
Fix bad lookups into the glossary
Switch ASCII art to the existing SVG
Update dispatcher actions
Improve timing warning message
Document the principles of a CI Loop
master: remove the need for the ERROR message
Expand job_details doc string with available keys
Replace 1.10 deprecated get_field_by_name with get_field method.
Add support for extra QEMU options
LAVA-747 - Report parameters as result metadata
Switch from using png to svg for the architecture diagram
Add a page to display the pipeline timings
Expand glossary and add sections on parameters
Add D03 device type to pipeline
Add an introduction to results in LAVA.
Add docs on the standard test jobs
Simplify and clarify the "first install" docs
LAVA-584 - expand for sub-types of device-types
LAVA-706 - codebase structure
Port submission priority support to V2
Adopt a similar contents layout to the django docs
Device dictionary output
Extend qemu jinja template for cortex-m3
Pipeline log: use a better icon for download
Update bootstrap from 3.1.1 to 3.3.7
Fix relative url for suite results
Add nbdroot as optional parameter to the schema
Update qemu/kvm templates to allow overriding of guestfs interface
LAVA-93 mark some roles as essential
lava-dispatcher (2016.11-1) unstable; urgency=medium
.
* New production release aimed at Stretch.
Release details available at:
https://projects.linaro.org/browse/LAVA-789?filter=12078
* Update compat
* Highlights of upstream changes:
Ensure namespace is available to the test shell.
cpio output is not an error
Downgrade message match failure to a warning
Fix up usage of LXC with a device
Only attempt lxc-destroy if the container exists based on exitstatus.
Drop ShellLogger __del__ command
Improve accepts for monitored QEMU
Strategy needs to check methods, not device_type
Support parallel (pbzip2) compressed bzip2 files
Allow vendor U-Boot builds using Ctrl interrupts
Improve cpio extract command line and error message
slave: do not send None as error code
slave: send a SIGINT to lava-dispatch to cancel
Making target IP and MAC addresses available in lava test shell
TestShell: allow multiple test shells
Fix kernel message detection for panic
testdef: remove unused common data
Fix the logic regarding the pipeline building
shell: make testdef_index common data a list
device/target: allow multiple boot control chars
Add 943907AEVAL1F to pyocd deploy supported devices
Check if container exists before destroying.
Do not generate results.yaml
Fix uefi_menu pipeline for fastboot.
slave: move tmp files to /var/
Add device to lxc only when it is available.
device-types: add Amlogic meson8b-odroidc1 board
LAVA-755 - use UUID for guest, not label.
slave: send an error code when err is not empty
Fix git-repos reference in inline test definition.
Allow a different bootloader interrupt character
Make pipeline logging python3 compatible
slave: fix a crash under python3
device-types: add Amlogic meson-gxbb-p200 board
device-types: add TI da850-lcdk
bootloader: add ramdisk_raw option
Detect invalid characters in testdefinition name
Fix timeout inheritance from job definition blocks
Set the right permis when creating the ramdisk dir
ci-run: exclude the .eggs directory
tests: Close the file descriptors
Fix root directory of V2 apache config
LAVA-760 - Trim LXC log output
Require a summary and description to validate
Additional changes for removable media usage
Fix extra_options to always be a list of strings
Do not set kernel type for all 'kernel' YAML keys
Action: re-raised the exception without modification
LAVA-748 ensure test definition names are unique
Change sample jobs urls to images.v.l.o
Allow testdefinitions without run steps
Generate the pipeline description on the slave
slave: fix the END message parameters
Use the QEMU extra options where available
slave: resend the END message if the ack was lost
slave: send the error file along with the END message
Move tmp files to /var/lib/lava/dispatcher/job_id/
Only extract the nfsrootfs once.
Use the uboot interrupt prompt from templates
Improve LXC HiKey support.
Don't end test shell at invalid testcase name
SignalMatch: simplify and fix exception message
Shell: improve report formatting
Shell: factorize handling of measurements
RetryAction: log errors
Shell: factorize signal handling
Move override and log_results functions into Action
Initial support for PyOCD
Use \r\n for newlines
Allow guest drive interface to be specified
allow-modify functionality added
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
The 2016.11 release is currently in final preparation and is targeted
at Debian 9 Stretch. We've tried to get the documentation and support
up to a point where users of Stretch have a usable platform before
later upgrading via stretch-backports.
I expect to get 2016.11 into jessie-backports about 5 days after that release.
A few reminders on how the Debian release process (which starts in
January 2017) will affect things, most of which are the same as during
the jessie release process. More details will be given on this list as
the Debian release process continues.
0: New lava-server and lava-dispatcher uploads to Debian won't migrate
into Stretch once the freeze starts. This means that these will not be
available in jessie-backports either as the presence of the same
version of the package in Stretch is required for an upload to
jessie-backports. I expect to get 2016.11 and 2016.12 into
jessie-backports.
1: Once Stretch is released, stretch-backports will be opened and
uploads to jessie-backports can resume. Debian makes the release when
the release is ready, so there are only estimates of when Stretch will
actually be released. The key information is that the freeze to
prepare the release starts in January 2017.
2: I expect to keep jessie-backports updated at least for the first
few releases. This list will carry more information about when there
may have to be changes in this area. Essentially it depends on
movement within Debian after the Stretch release.
3: During this period, uploads will remain available on
images.validation.linaro.org built for jessie - lava-dispatcher will
only be available for amd64 from those repositories. If you are using
armhf or arm64 for the dispatcher, please let us know. Use the
production-repo to get updates of lava-server and lava-dispatcher
during the Stretch release process until jessie-backports is open
again.
4: At some point after the release, the production-repo will start to
be built on and for Stretch. stretch-backports and jessie-backports
will need to be open before this change is made. More details of that
will appear on this list before it happens.
5: Once Stretch becomes Debian stable, Jessie becomes Debian
oldstable. Debian testing gains a new name: Buster. Get used to the
names.
6: It is worth checking *now* that all your apt sources on all your
machines (including VMs) specify jessie and not stable. Otherwise, as
soon as you do an apt upgrade after the Stretch release, your machine
will start the upgrade from jessie to stretch. Make sure you are ready
for that upgrade and that it doesn't happen until you are ready for
it. Also check the images you use in test jobs and the build tools you
use to prepare those images.
7: We have been building images based on Stretch ahead of the release
for test jobs using QEMU and similar. We have also updated the
documentation in the next release to use stretch images.
8: At some point after the Stretch release, there will come a point
where support for jessie is dropped and uploads to jessie-backports
will cease. The upgrade to stretch will be required at some point to
continue getting updates to LAVA. Prepare for it ahead of time and
everything will go more smoothly. Further details of the timing will
be announced on this list.
9: The LAVA software team have been testing with stretch for some time
and will continue to do so. We do not expect any issues with LAVA
packages when upgrading from jessie to stretch. Nonetheless, all
admins are recommended to prepare for the new Debian release and to
perform the upgrade under their own control. Make sure your other
services will work after those machines upgrade to Debian 9 Stretch.
More information: https://release.debian.org/stretch/freeze_policy.html
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
$ sudo apt-get install -t jessie-backports python-sphinx
To solve some problems with the build of the documentation, we have
had to move to the version of python-sphinx available in
jessie-backports. (Essentially, our documentation is now so large that
it is causing problems for the theme we've chosen and we've had to
make some changes to keep improve the navigation within the pages.)
If you are building lava-server in sbuild, there is an addition option
required to sbuild:
--build-dep-resolver=aptitude
The build script in the lava-dev package uses this option already.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
There have been problems with using uWSGI for lava-server, including
preventing updates of 2016.6-2~bpo8+1 in jessie-backports.
The LAVA software team have completed a change which will be included
in the 2016.11 release which removes the need for the
libapache2-mod-uwsgi, libapache2-mod-wsgi and uwsgi-core packages. The
replacement is the gunicorn package.
This does involve changes to the apache configuration of all sites
which upgrade to 2016.11.
Ahead of installing 2016.11, two apache modules need to be enabled.
https://staging.validation.linaro.org/static/docs/v2/installing_on_debian.h…
$ sudo a2enmod proxy
$ sudo a2enmod proxy_http
$ sudo service apache2 restart
These will not affect older installations.
Then install 2016.11 and your instance will continue to operate.
If you have made changes to
/etc/apache2/sites-available/lava-server.conf then the changes to the
upstream file will need to be merged into your changes.
If you choose to remove libapache2-mod-uwsgi after completing the
install, be aware that there is a Debian bug which may affect you.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=839666
If your apache server fails to restart after removing the
libapache2-mod-uwsgi package, you will need to workaround this bug by
executing:
$ sudo a2dismod uwsgi
$ sudo apache2ctl restart
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
2016.9 includes a rewrite of each of the daemons used by LAVA,
lava-server, lava-master and lava-slave. When installing the upgrade,
admins need to accept the maintainer's version of the scripts in
/etc/init.d/ for these daemons. Where previously changes were made in
the scripts in /etc/init.d/, the same parameters can now be changed in
associated configuration files to prevent these conflicts in the
future.
/etc/init.d/lava-slave - change /etc/lava-dispatcher/lava-slave
/etc/init.d/lava-master - change /etc/lava-server/lava-master
/etc/init.d/lava-server - change /etc/default/lava-server
The prompt would look something like this:
Configuration file '/etc/init.d/lava-server'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** lava-server (Y/I/N/O/D/Z) [default=N] ? y
See https://validation.linaro.org/static/docs/v2/pipeline-server.html#configuri…
References in the changelogs, commit history and announcements which
start with LAVA- followed by a number relate to the planning stories
which the LAVA team use for new features. The LAVA planning has now
been opened up to anonymous access by appending the reference to a
stub URL: http://projects.linaro.org/browse/, e.g.
http://projects.linaro.org/browse/LAVA-62
The planning stories do not support comments or changes other than by
the LAVA software team. If you have comments on the stories, please
use the lava-users mailing list.
lava-server changes
Show measurements and units of testcase detail page
Drop lxc schema - no support for alternatives
Configure logging for linaro_django_xmlpc calls
Fix link to lava-tool context-help
Fix the "Debian-based distribution" link grammar
Improve monitor test type, allow multiple tests
LAVA-494 allow searching for device tags in tables
Add support for LXC with multinode
Fix broken ref link for unit_test
Fix NoReverseMatch from ImageReports2.0 editing.
Fix typo in ./share/validate.py --hostname
Update hikey jinja2 templates for LXC
Fix occasionally failing multinode tests.
Add a configuration file for lava-master
Update lava-slave and lava-master docs
LAVA-374 - drop versiontools
Tidy up some build errors and reformat consistently.
Improve documentation describing inline test definitions
Major improvements for the "Writing MultiNode" page
Add docs for the lava-vland-names change
Using BOOTX in Juno bootcmds
Correcting default uboot commands in juno device template
Document the code locations for developers
Ensure multinode description is generated correctly
Allowing overrides to connection and action timeouts
master: Fix log rotations
Remove filtering by filter id for available testcases in image reports.
Init: remove the right lock file and simplify
Adding uboot jinja2 templates for all Juno flavours
LAVA-739 - declare sub_id and job.id for multinode
master: don't validate jobs on the master
select_device only accepts pipeline jobs
Remove legacy SyntaxHighlighter from job definition pages.
lava-master: simplify log file handling
Fix title disappearance on image reports.
Ignore retired devices in validate call
Don't add anchors to logs for running jobs
Don't crash when log contains invalid test names
Schema: accept 'parse' for inline test definitions
Use libYAML when loading logs
LAVA queries use cases.
Bug link fix for url regex.
Add LogEntry for BugLinks.
Update the init scripts
Remove the entry point as it's no longer needed
Ensure test-case name is valid
Extend vland unit tests to cover assignment
Expand on the device dictionary and templates
Expand the notes on using jessie-backports
Add a Contents page
Rewrite lava-server command line from scratch
events: set umask to a restrictive value
Bug links port.
LAVA-200 - track admin actions made in the UI
Add device tags support to V2
Allow the monitor test type
Update qemu command line options for arm64
lava-dispatcher changes
Catch the edge case where serial line corruption means we miss the testrun
Add support for alternative Test actions
Update manpage of lava-slave for configuration change.
Allow configuring the slave from file
git: use -C instead of --git-dir
Allow setting the envrionment on most systems
Handle exceptions from subprocess from compression
download: don't expand the path when retrying
Grab finalize from the root pipeline
Log exception as strings and not objects
LAVA-740 - export vlan names and interface names
Reopen log file when rotating the logs
LAVA-699 - Support hi6220-hikey board with lxc
Init: remove the right lock file and simplify
pipeline:vland: raise JobError if switch_id or port_id is None
Remove redundant assignation
Allow logging at the validate stage
Force the shell to produce an initial prompt
Handle errors if the testrun start is omitted.
Pipeline parser: sort and reverse in one call
logs: remove unused support for local logs
slave: put all tmp files into the same directory
Power: improve logging message
Exclude patterns from description
Port lava-slave to Python3
ApplyOverlayTftp: remove duplicated untar_file
ExportDeviceEnvironment: only source valid shell_file's
add LEDE to available distributions
Always log the command to be run beforehand.
There is currently a delay on uploading lava-server to
jessie-backports due to problems with uwsgi in Debian. A backport will
be made available once these problems are fixed. In the meantime, the
same build as will later appear in jessie-backports is available in
the production-repo:
https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…https://tracker.debian.org/pkg/lava-serverhttps://tracker.debian.org/pkg/lava-dispatcherhttps://qa.debian.org/developer.php?email=pkg-linaro-lava-devel%40lists.ali…
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
A number of the configuration files which manage the daemons used by
lava-server and the lava-slave daemon for lava-dispatcher are being
updated in 2016.9. This has implications for those who install the
upgrade, especially if this is normally done by automation.
Configuration file '/etc/init.d/lava-slave'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** lava-slave (Y/I/N/O/D/Z) [default=N] ? y
Installing new version of config file /etc/init.d/lava-slave ...
dpkg will check the checksums of the init files against the current
and previous packages and halt installation with the above prompt in
case of doubt. Admins may need to install the update manually
(directly on the command line).
The recommendation is to accept the package maintainer's version but
all admins may want to check the difference between the two.
Changes in this release include fixing the log rotate handling of
lava-master using reload and a rewrite of the lava-server CLI script
to drop the use of entrypoints which changes the syntax of the daemon
options.
The most common reason for init files to be modified is to change the
loglevel. This can be done in /etc/default/lava-server:
# Additional options that are passed to the Daemon.
DAEMON_OPTS="loglevel=debug"
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
LAVA has long been developed in the open as far as the git
repositories and bug reports are concerned. However, the git commit
history can often include references to LAVA-NNN and these references
have previously been obscure.
Anonymous access has now been granted to all LAVA-* references, so
here's some context on what and where.
https://projects.linaro.org/projects/LAVA
This is a JIRA instance and we refer to issues in JIRA as cards or stories.
All stories in LAVA are now visible to the community, so you can map
any of the LAVA-NNN references directly to the actual URL by
prepending projects.linaro.org/browse/LAVA-NNN, e.g.
https://projects.linaro.org/browse/LAVA-734
It has not been possible to open the actual organisation of the
stories (KANBAN) and some elements do not provide the same links as
when viewed internally but hopefully some of those issues can be
addressed in the future.
For now, community members can follow any explicit LAVA-NNN reference
from the git history or elsewhere to the actual story and view the
complete description and all comments on that story. Most stories also
provide a direct link to the review which implemented that story
within the Comments section. Merged reviews provide information in the
"Included In" menu about which branches and tags include that commit.
Tags match the production release of the same name as the tag, so this
allows everyone to know which releases contain which changes.
Stories which are currently status Open may simply be ideas which
still need to be scoped. Some of these will be dropped without being
implemented as use cases and code development moves on.
Dates in stories are very rough estimates, usually longer than it will
actually require.
If community members have comments or questions on particular stories,
please use the lava-users mailing list.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
This release moves to requiring django 1.8 from jessie-backports.
(django1.7 is no longer supported, django1.8 is the LTS release)
See https://lists.linaro.org/pipermail/lava-announce/2016-June/000010.html
As part of this release, OpenID and therefore Launchpad login support
has been removed - the openid support for django cannot support
django1.8.
See https://lists.linaro.org/pipermail/lava-announce/2016-June/000011.html
This release also includes support for running lava-server with the
recently released django1.10 which will soon be available in Debian
Stretch (current testing).
2016.8 includes a rewrite or large sections of the documentation and
the new logging support for LAVA V2.
2016.8 includes the changes planned for 2016.7, so this is a larger
set of changes than previous releases.
lava-server changes
lava-master: use also get_env_string for multinode
Handle unrecognised result messages.
Publisher: drop privileges at startup
Add lava-publisher init scripts
Update mustang jinja template
Similar jobs feature.
Django1.10 fixes
v2: include a 'secrets' field in the job def
Adding device-type templates for juno
Fix a deprecation warning with render_to_string
events: add more details and use a useful username
Ensure failed health checks go directly to offline.
Faster loading of yaml logs
Add a u-boot-commands timeout just for panda
Improve error handling in result metadata
use job.id inside a not job.is_multinode conditional
Fix multinode link from definition back to the job.
Allow parentheses in test case names
Allow the d02 debian installer grub device to be overridden
in device-dictionary
Allow for creating devices already offline.
Device state transition validation.
Open context-sensitive help in a new browser tab
Unavailable qemu command should not fail unit tests.
LAVA-719 - support branding of source and bugs URL
Ensure logging to django logs is info or higher
Fix e1d66f to use pk when not multinode.
Create and display measurements with units
result: don't crash when parsing an invalid result
Implement notification blacklist.
results: handle skip result
Simple notification list.
Add 'name' to testcase export.
Use the right syntax for character delays
First device configuration for ST b2120h410
Fix bug #2278 - inconsistent multinode job id / alias usage
Fix HTTP500 by allowing for + in test case names
Show job sub_id for multinode jobs.
Implement IRC notifications.
Fix bug #2263 - parameters and params reference in job def and test def
LAVA-708 - Device path should be a list
Fix a crash when viewing a query for the first time
result: show the metadata as a list (and sublists)
Rename conflicting notification properties.
log: don't show 'extra' result data
log: add a link to each line using AnchorJS
log: skip broken strings
log: add an icon for the download button
log: add link from the result page back to the log
Results: improve admin page
TestResults: order by job_ids then name
Simplify a bit the result page
Remove unnecessary loading of django-tables
log: redirect complete_log to the job_detail page
log: add a link to the result page for each result
log: fix HTML syntax errors
log: improve rendering of errors and exceptions
Update load addresses for larger multiplatform kernels
Fix result table
log: fix a bug when the page is reloaded
Protect from admin error in health check submission
Fix default value for device_path to be None and not 'None'.
Add missing device_path to nexus jinja templates.
log: adapt the result parser to the new log stream
mustang UBoot needs 32bit header
logs: update job status and device information
log: change the arrow when clicking on the affix
Fix handling of context with multinode
Fix metadata handling for multinode and dynamic connections
Fix hidden-device-type listings in JobTable
job: remove redudant information
Events: add a monitoring thread
Initial notifications for v2.
lava-master: save the logs in output.yaml
job: add a new template for the new log format
LAVA-262 Allow admins to expire user accounts
log: better formatting of tracebacks
Remove support for Django < 1.8
Improve scheduler debug with device details.
.
Documentation updates
Add links and notes to developer branch guide
Add notes on making Lava Test Shell portable
Add notes on running lava-server unit tests
Add timeout documentation.
Update the developer guide
Document the 'secrets' dictionary
Ensure V2 documentation examples are available.
update local user account image
tidy up api docs
Remove multinode use cases
tidy up the writing-multinode page
expand simple-admin for admin roles
tidy up hidden toctree listings for previous/next markup
Update chapters for theme
Switch to the bootstrap theme
updates for multinode and simple administration
Major update to the docs for writing multinode tests
move all examples into one directory and add test definitions
move lava tool issues to a separate file
fold the FAQ into the lava-tool docs
update the multinode use cases
port the mustang example to a separate yaml file
use rst macros for see also
Add publishing API ref doc
initial content for a results intro
Move doc yaml to a directory
WIP rewrite of the multinode doc
Start thinking about how to grow a lab
Re-org some early admin stuff
Split out the completed YAML jobs
Query omit documentation updates.
Fix documentation for test definition name handling
add instruction for -t jessie-backports
move example YAML to an rtsi for easier checking
add notes on setting up the first device and device type
fix whitespace in migration example
Update the scheduling ordering with links
Add notes on LAVA being developer focused
Update other examples for deploy change
fixup deploy action
add example of first qemu V2 device
start the pipeline design page
Minor wording tweaks
Rework the hacking session doc
expand notes on first installation
tweaks and updates for writing tests
Fix definition link to log for pipeline
Updates for test repositories
update multinode docs for V2
fix build messages and errors
update examples of params support and custom scripts for parsing
complete fixme in advanced-installation
add background on CI and LAVA
add notes and images for first job submission and results
explain the first job and tidy up the example YAML
Clean up health check docs
add notes for first job
Significant cleanup of wording around lava-test-shell
Add lots of code-block:: yaml directives
Add details of features and architecture.
Add content to the what-is section
lava-dispatcher changes
Only require conversion parameters if using conversion.
Support some basic kernel conversion tasks
Export the 'secrets' dictionary to the overlay
uboot: only log self.errors when it's not empty
Enabling NFS only deployment
Fix connection_timeout handling for lava-test-shell
validate needs to set errors, not raise
Fixes for multinode without needing ACK
Device environment: fix a crash if env_dut is None
Unavailable qemu command should not fail unit tests.
Use character_delays instead of character-delays
Tests: Use the name defined in the job definition
Show when a revision is being applied for VCS
Fix bug #2263 - parameters and params reference in job def and test def
Improve logging and use lazy logging
Log more test results metadata
LAVA-708 - Device path should be a list
Support kickstart installations on grub, and centos distro type
Allow for a prefix with nfsrootfs
Finalize: send the job status with the right level
log: send the full exception untouched
beaglebone-black: update load addresses for larger multiplatform kernels
Take into account, device_path that is unset.
Fix YAML formatting of wait string log
change the result logging format
Add sample nexus9 device dictionary.
LAVA-701 - Support vendor flashing using fastboot.
utils/messages.py: fix out of index error
Add support for fixup dictionary and patterns
Ensure guest command is available in multinode
Stop V2 test shells needing an ACK from the dispatcher
logging: add datetime and a better structure
Upgrade the failure test case for lavabot
First batch of VExpress changes
Handle ValueError while waiting
To enable jessie-backports, simply copy your existing apt source for
jessie and change jessie to jessie-backports, then run apt update.
Installations from backports are not automatic, you'll need to tell
apt to select jessie-backports for django1.8, as outlined in previous
mails to this list:
$ sudo apt install -t jessie-backports python-django
python-django-common python-django-tables2
Once 2016.8 migrates into Stretch and is then backported to
jessie-backports itself, apt will be able to handle the extra
dependencies.
https://tracker.debian.org/pkg/lava-serverhttps://tracker.debian.org/pkg/lava-dispatcherhttps://qa.debian.org/developer.php?email=pkg-linaro-lava-devel%40lists.ali…
The production repo also has this release.
https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…https://validation.linaro.org/static/docs/v1/installing_on_debian.html#lava…
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Due to resource shortages in the team, conference time and a few
problematic fixes which still need testing, we will have to skip the
delayed 2016.7 release and move all changes to the 2016.8 release, due
at the end of the first week of August 2016.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
OpenID has continued to be available for lava-server on jessie as long
as lava-server stayed on django1.7 but the OpenID django support does
not operate with django later than 1.7. To continue with upstream
django security and bug fix support, lava-server needs to move to
requiring django1.8 which means that OpenID can no longer be supported
in jessie. Support for OpenID in unstable and testing was removed when
django1.8 support first arrived.
Once lava-server 2016.7 arrives in jessie-backports, it will conflict
with the python-django-auth-openid package - this means that to
install the 2016.7 upgrade from jessie-backports, apt will first cause
the removal of python-django-auth-openid. (Allowing
python-django-auth-openid to remain on the system will corrupt
subsequent django operations, causing the package installation to fail
during the database migrations.)
OpenID support in lava-server is also due to be removed for 2016.7, so
the documentation and settings will no longer reference OpenID.
LDAP support, Debian SSO and local django accounts will remain available.
https://packages.debian.org/stable/python-django-auth-openidhttps://wiki.debian.org/DebianSingleSignOn
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Further to the notice about the availability of django1.8 support:
https://lists.linaro.org/pipermail/lava-announce/2016-April/000007.html
The 2016.7 release of lava-server will be moving to requiring
django1.8 from jessie-backports and will no longer work with django1.7
available in jessie.
This has been done because the 1.7 tree is no longer receiving
upstream security or bug fix support.
lava-server installations newer than 2016.2 can use django1.8 already,
as long as python-django-tables2 is updated at the same time. So you
are invited to upgrade to django1.8 at any time prior to upgrading to
2016.7.
To add jessie-backports, copy the apt source for your favourite Debian
mirror and paste that as a new line with jessie updated to
jessie-backports.
e.g. for the http://mirror.bytemark.co.uk/debian mirror:
deb http://mirror.bytemark.co.uk/debian jessie main
deb-src http://mirror.bytemark.co.uk/debian jessie main
becomes
deb http://mirror.bytemark.co.uk/debian jessie main
deb http://mirror.bytemark.co.uk/debian jessie-backports main
deb-src http://mirror.bytemark.co.uk/debian jessie main
$ sudo apt update
$ sudo apt install -t jessie-backports python-django python-django-tables2
$ sudo apache2ctl restart
$ sudo service lava-server restart
It is imperative that python-django-tables2 is updated at the same
time as python-django - the old version of django will not work with
new tables and the old version of tables will not work with new
django.
Backports do not automatically replace packages you already have
installed, so you remain in control of which backports get installed
onto your systems and can run apt dist-upgrade without getting extra
upgrades.
Backports related to lava-server are also fully supported and are the
recommended (soon to be required) way of installing updates for
lava-server.
After 2016.7, updates of lava-server will not install on jessie
without django and django-tables being updated from jessie-backports.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
lava-server changes
* Update V1 docs for Ubuntu changes - lava-server no longer
migrates into Ubuntu and was removed from Xenial.
* Drop heartbeat support
* Prevent scheduler ValueError in reservation
* scheduler: reduce the number of SQL queries
* Expose DISALLOWED_USER_AGENTS to handle search bots
* Add a page for listing Pipeline Devices.
* Add Auth support in REST API for more functions
* Remove device status glyphicons everywhere, since heartbeat is dead.
* Create metadata on the number of test definitions
* Remove the need for extensions
* Remove deprecated lava_projects
* Update docs for guestfs and resulting issues.
* Enable job definition metadata.
* dispatcher-master: support zmq CURVE encryption
* Add documentation on using ZMQ curve
lava-dispatcher changes
* Load the overlay as an extra QEMU drive to prevent
need for loopback devices.
* Support delays between sending characters for all actions
* LXC support for fastboot and adb and drop the need for adb or
fastboot to be installed.
* Remove adb related code
* Port V2 unit tests only to python3
* ZMQ: Support encryption of the logs sent to the server
* Add support for QEMU Debian Installer tests
* Update standards version (no changes)
* Remove adb and fastboot from recommends and add python-guestfs as a
dependency.
To use LXC with LAVA 2016.6 (V2 only), the lxc package itself needs to
be installed from jessie-backports (the lava-dispatcher package has a
version requirement in the Recommends which enforces this, if lxc is
installed). Additionally, for the new version of LXC to work
correctly, the dispatcher itself needs to be rebooted to apply the LXC
updates to the kernel boot arguments.
To enable jessie-backports, simply copy your existing apt source for
jessie and change jessie to jessie-backports, then run apt update.
Installations from backports are not automatic, you'll need to tell
apt to select jessie-backports for the LXC package:
apt install -t jessie-backports lxc
https://validation.linaro.org/ has already been updated to 2016.6.
Packages have been uploaded to Debian (unstable) and will appear on
the mirrors in due course. Once these versions migrate into stretch,
backports will be made to jessie-backports.
https://tracker.debian.org/pkg/lava-serverhttps://tracker.debian.org/pkg/lava-dispatcherhttps://qa.debian.org/developer.php?email=pkg-linaro-lava-devel%40lists.ali…
The production repo also has this release.
https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…https://validation.linaro.org/static/docs/v1/installing_on_debian.html#lava…
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
The 2016.6 release which is currently in preparation adds some
important dependencies which some admins may prefer to install ahead
of the upgrade.
Debian has the concept of Dependencies which must be installed and
Recommends which are optional but expected to be useful by most users
of the package in question. The Recommends for GuestFS can be omitted
from the installation if admins desire but this needs to be done ahead
of the upgrade to 2016.6.
V2 in 2016.6 adds support for GuestFS which removes the need for
loopback support for QEMU devices. GuestFS supports a wide range of
filesystem utilities and other virtualisation support tools, not all
of which will be of interest to LAVA. (Notably, libguestfs Recommends
a mail-transport-agent and the default MTA in Debian is exim. With the
default exim4 config, this simply adds a local MTA with no external
support.)
Debian supports opting out of Recommends when installing packages, so
if admins have concerns about extra packages being installed on the
dispatchers (e.g. if using ARMv7 dispatchers or simply to reduce the
complexity of the install) then Recommends can be omitted for the
installation of these dependencies:
The option is:
$ sudo apt --no-install-recommends install python-guestfs
We'll add a note to the installation docs for this too.
With the move to V2 having a dumb dispatcher, the dependencies
themselves do need to be installed to allow the server to dictate what
jobs the dispatcher can run without the dispatcher failing due to
missing support. If later LAVA upgrades need any of the packages that
would have been installed as Recommends, LAVA will add an explicit
dependency on just those packages and any others will remain optional.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Updates of existing backports are not usually announced in this way
but there is a note to consider, so I'm covering that here.
Independent of the update of the lava-server and lava-dispatcher
backports to 2016.4, admins using jessie-backports do need to be aware
of a issue which is still pending related to django and django-tables.
A backport of django-tables is pending, waiting for these two packages
to also be accepted into jessie-backports:
https://ftp-master.debian.org/new/django-haystack_2.4.1-1~bpo8%2B1.htmlhttps://ftp-master.debian.org/new/python-fudge_1.1.0-1~bpo8%2B1.html
These are required before django-tables can build inside
jessie-backports and therefore before I can upload a build of
django-tables for jessie-backports.
Irrespective of the version of LAVA being used, if django itself is
installed from jessie-backports, the version of django-tables in
jessie will cause a HTTP500. The fix is to either downgrade django
back to the version in jessie or to pull in the version of
python-django-tables2 from *testing*.:
https://packages.debian.org/stretch/python-django-tables2. It is this
upstream version which is to be backported to jessie-backports.
Once these last two dependencies clear the queue for backports, I'll
make the backport of django-tables and then admins can have django and
django-tables from jessie-backports which will work. i.e. the
installed django needs to match up with the installed django-tables -
either both from jessie or, once available, both from
jessie-backports.
This isn't a problem in LAVA as such, it affects LAVA and I'll have
the fix available just as soon as the queue clears within Debian.
Finally, LAVA is likely to move to using django from jessie-backports
once this issue is fixed so that instances gain the security fixes in
the django 1.8 release - at that point, LAVA will ensure that the
updated django-tables is pulled in at the same time. LAVA has been
running with django 1.8 (and django1.9) for developers for some time,
testing with 1.8 will continue in the meantime.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
lava-server changes:
Add support for python-django-debug-toolbar
Deleting V1 filters now cascade delete image chart filters.
Reduce the number of SQL queries used on common pages.
Improve scheduler iterative performance.
Add support for deleting unused tokens
Stop runaway healthchecks in V2.
Migrate option_list to argparse for django 1.8 and later.
Allow authentication with result export in V2
Drop references to Ubuntu beyond 2016.9.post1
Implement omitting individual results from queries in V2
Indicate omitted results and allow including them back in.
Add a management command for refreshing queries
Change V1 measurement field to be float only.
Clean up top-level documentation
Introduce limit to queries in V2.
* Suggest python-django-debug-toolbar
* Refresh all V2 queries during package postinst to ensure
materialized views are available.
lava-dispatcher changes:
Allow for adb not being available on some arches
Support ramdisks that are not compressed
Move the ramdisk header requirements into the device template
Fix calling the Bzr unit tests
Use tar flags from deployment data when unpacking overlay
kvm-aarch32: introduce device type for 32-bit guest on 64-bit KVM
host
kvm-aarch64: expose a virtual random number generator (RNG) to the
guest
Provide an apache2 config for V2 workers
Initial support for device using GRUB bootloader in V2
Add in the ability to halt the V1 master image before powering off
No longer require ssh options or identity_file
Don't use -p port in SCP options
https://validation.linaro.org/ will be updated in due course (probably
at the start of next week).
Packages have been uploaded to Debian (unstable) and will appear on
the mirrors in due course. Once these versions migrate into stretch,
backports can be made to jessie-backports.
https://tracker.debian.org/pkg/lava-serverhttps://tracker.debian.org/pkg/lava-dispatcherhttps://qa.debian.org/developer.php?email=pkg-linaro-lava-devel%40lists.ali…
The production repo also has this release.
https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava…https://validation.linaro.org/static/docs/v1/installing_on_debian.html#lava…
Information will also appear at
http://releases.linaro.org/components/lava/ in due course.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
The 2016.2 release marks the start of the migration to the pipeline
dispatcher. The previous dispatcher code, the documentation for that
code and the support for JSON job formats are now *deprecated*. As a
result, the corresponding lava-server code has also been *deprecated*,
this includes Bundles, Filters and Image Reports (original and 2.0).
The deprecated Dashboard is replaced by Results with Queries and
Charts which have been ported to the data produced by the pipeline
testjobs.
Future development within LAVA will be based solely on the pipeline
dispatcher and associated server side objects. Support for the
deprecated dispatcher will be maintained for future releases only
during the migration and *will be removed* once the migration is
complete. The migration is expected to take most of 2016. Individual
devices and particular items of job support will be gradually disabled
until the migration completes for all devices and all jobs.
The migration process involves:
0: Adding pipeline support for devices and deployment methods
1: Migrating user submissions and automated submissions to the
pipeline support. For the Linaro LAVA instances, this will happen
within the particular teams inside Linaro.
2: Making selected devices exclusive to the pipeline so that JSON jobs
can no longer be submitted for those devices. At this point, support
for JSON jobs which rely on those devices will cease.
3: Once all devices are exclusive, reject all JSON submissions.
4: Disable the old scheduler daemon on the LAVA instance.
At some point, probably in 2017 - a month or two after JSON
submissions cease - the migration will complete by executing:
5: Removal of code support for the old dispatcher in lava-dispatcher
and lava-tool codebases and associated documentation.
6: Removal of code support for database objects specific to the old
dispatcher, like Bundle, in lava-server with associated documentation.
This will involve the deletion of the Bundle data as well as image
reports, filters and BundleStreams. TestJob objects which are not
pipeline jobs will also be deleted.
7: Removal of the rest of the code which is still dependent upon or
only used to support deprecated objects and functions.
8: Modification of code which only exists to isolate the deprecated
objects from the pipeline objects. e.g. newly created jobs or devices
will default to pipeline support.
LAVA instances outside Linaro will need to manage their own migrations
during 2016 if updates are to be applied during 2017.
It is not possible to retain the database objects without the
deprecated code, so owners of individual instances may choose to
create an archive instance which has no online devices, accepts no
submissions, just provides access to a snapshot of the data at the
time that JSON submissions ceased. To maintain access to the archived
data, this instance must not upgrade to LAVA releases after the
archive is created and the original devices should persist in the
database but be kept in the Offline state. Devices should also have
submissions restricted to a single administrator account for which no
API token exists.
More information on the details of the migration and the state of
support for jobs using the old dispatcher will be announced on this
mailing list.
As a reminder - lava-announce is a read-only list. Posts to this list
are only made by the LAVA software team. Replies need to be directed
to
linaro-validation(a)lists.linaro.org - your email client should do this
for you, using the Reply-To header added by mailman.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Please reply to linaro-validation(a)lists.linaro.org
Please let us know if you are using OpenID authentication with LAVA.
Newer versions of django will make it impossible to support
django-openid-auth in Debian unstable and testing. The version of
django-openid-auth in Jessie can continue to be used, so we would like
to know how many users want to continue with this support.
OpenID as a protocol has been dying for some time and Linaro has moved
over to LDAP, which is fine if LDAP is already available.
The time pressure for this change is coming from the schedule to get
the latest django and the latest lava packages into Ubuntu Xenial
16.04LTS which means that support needs to be implemented in the
2015.12 or 2016.1 LAVA releases. This is why this is quickly following
the trusty change. We have been aware of the issues with
django-openid-auth for some time, it was only when we had completed
the move of the Cambridge lab to LDAP that changes involving
django-openid-auth could be considered.
If you are using OpenID authentication (e.g. using Launchpad or Google
OpenID), please let us know.
If you would like to see some other forms of authentication supported,
also let us know. We can investigate Python Social Auth
(http://psa.matiasaguirre.net/), if there is interest.
If we don't hear from users who want django-openid-auth support for
use on Debian Jessie, we will drop django-openid-auth support from all
lava builds. This will leave LDAP and local Django accounts in
2015.12.
If anyone has experience of other django authentication modules, also
let us know.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
See also:
https://lists.linaro.org/pipermail/lava-announce/2015-November/000002.html
which was also sent to these lists (except linaro-dev).
So far, nobody has come forward as a Trusty user. The only Trusty
instance of which we are aware is already due to migrate to Debian
Jessie.
The LAVA software team are now applying updates which will freeze LAVA
software support for Ubuntu Trusty at 2015.9 for lava-dispatcher and
2015.9.post1 for lava-server due to the complexities of supporting
both django1.6 and the current django1.7 in Jessie and django1.8,
possibly django1.9 by the time Debian Stretch is released.
The last packages for Ubuntu Trusty 14.04LTS will be:
lava-server 2015.9.post1
lava-dispatcher 2015.9
Once these changes are applied, the Debian packaging used to build
future versions of LAVA packages will prevent builds against django1.6
and prevent installation if django1.6 is found, in order to prevent
database corruption.
This means that Trusty users will not be able to use the results of
the dispatcher refactoring.
Ubuntu Xenial Xerus - which is planned to be the 16.04LTS in April
2016 - is expected to pick up LAVA software releases from Debian up
until the 2016.1 release (possibly 2016.2) and is also expected to be
using django1.8. The next Debian stable release (Stretch), for which
no date has yet been set, may use django1.9.
Initial attempts at migrating a test instance from Trusty to django1.7
did not go well and the migration from Trusty to Xenial cannot be
supported by the LAVA software team - the recommendation is to go
directly from 2015.9 on Trusty to the same version available for
Debian Jessie but there will still be work to be done to prepare and
implement the migration which will be instance-dependent.
Documentation is being added to assist with this migration but there
will remain risks of data loss which will need to be managed for each
instance. It is imperative that anyone using Trusty has an up to date
backup of the postgresql database dump before considering any
migration. If the existing data is to be dropped, a new install on
Debian Jessie is recommended.
It is not possible for the LAVA software team to support all versions
of django from 1.6 to 1.9 - particular problems are known when going
from django1.6 to django1.7 as the methods to migrate the lava-server
database changed fundamentally in django1.7.
Notes are being added to the documentation on the trusty branch based
on 2015.9 to be released within lava-server 2015.9.post1 and to the
documentation in the master branch (which will go into 2015.12).
All future builds of LAVA software will now be made and uploaded only
to Debian and releases.linaro.org.
So far, nobody has come forward who is willing to maintain packaging
for LAVA software on any distribution other than Debian. As the
refactoring proceeds, we expect that it will become easier to package
LAVA for other distributions but the migration to the refactoring must
be complete first.
Everyone interested in or using LAVA is encouraged to subscribe to the
lava-announce mailing list which is low volume and only used for
substantial changes like this.
https://lists.linaro.org/mailman/listinfo/lava-announce
See also https://validation.linaro.org/static/docs/support.html
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
This is a call to *all* users of LAVA on Trusty - please let us know
who you are, what you're needs are and whether you are able to migrate
to Debian Jessie instead of going from Trusty 14.04LTS to Xenial Xerus
16.04LTS.
If you are using LAVA on any release of Ubuntu older than Trusty, the
only advice from the LAVA team is to immediately migrate to Debian
Jessie.
If you are using LAVA on Wily Werewolf or Vivid Vervet, you should
also consider testing the migration to Xenial and comparing with a
migration to Debian Jessie, as below.
LAVA is looking at a fix for the 2015.9 release but Django1.9 is in
beta release already. There are issues with django1.8 which are in
development. Currently, we are proposing that this update will be
applied to 2015.9 and made into a frozen release.
The master branch of LAVA will continue to develop and will need to
use more features only available in Django1.7 and later. Documentation
relating to installing Trusty would then be deprecated and removed in
subsequent releases from the master branch. Equally, future database
migrations on the master branch would no longer provide south support,
using the django migration support provided by django1.7 and later. So
these releases would not be built for Trusty - leaving only the frozen
branch.
Due to the complexity of supporting django1.6, it is unlikely that
updates will be available for the frozen branch once this happens..
The changes in the lava-server due to the ongoing refactoring will
mean that users of Trusty will be unable to migrate to pipeline
support until the server has also been migrated to Xenial 16.04LTS.
In addition to this, there is concern that migrating from Trusty and
django1.6 all the way to django1.8 or possibly django1.9 in Ubuntu
Xenial 16.04LTS is going to be problematic and the LAVA team will be
unable to assist in most cases.
The alternative is for someone with a reasonably complex lab running
Ubuntu to take up a role as tester of the frozen branch *and*
responsibility for patches which can maintain trusty support and
migration to Xenial 16.04LTS. The problem then will be that it will be
a very large transition when 16.04LTS actually becomes available -
only for the same lag to start all over again.
I'm unsure when Xenial will close the window for migrations from
Debian into Xenial - I expect that the 2015.12 release of LAVA will
migrate, I expect that 2016.1 will migrate too but I cannot be sure
about 2016.2 or 2016.3. That migration is completely outside the
control of the LAVA software team.
https://launchpad.net/ubuntu/xenial/+source/lava-server
Everyone considering staying on Ubuntu is advised to try a migration
to Xenial *now* - in a VM, with and without a recent backup of your
database and logs. Xenial currently has 2015.11. Also compare with a
migration to Debian Jessie by dumping and reimporting the database. In
each case, ensure that the permissions on /var/lib/lava-server and
sub-directories are retained from the original.
Please talk to us and test out what you are going to do.
https://validation.linaro.org/static/docs/support.html
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
It has become impractical to maintain support for the latest production
releases on Ubuntu Trusty 14.04LTS due to behaviour required by the
server side changes needed by the dispatcher refactoring. These are
important changes and are impractical to backport to the version of
python-django available in Trusty.
I have tried working around the issues but it involves pulling more
dependencies from Jessie or Vivid and in my tests, the existing
installation became confused and failed to install due to problems with
the database migrations. The LAVA software team do not have the
resources to maintain backports of the complete dependency chain in
Ubuntu, it is a much more manageable set in Debian Jessie (where the
same migration went without issues on multiple machines).
The changes in 2015.11 are almost exclusively related to the dispatcher
refactoring and therefore there is expected to be limited impact on
current installations on Trusty.
For the next release, we will investigate code changes which omit the
new support for Trusty builds but this will have the effect of
preventing those installs from working with the pipeline and the new
YAML job submissions. The documentation for Trusty will also be
updated, if necessary as a hotfix to 2015.9 just for Trusty.
If the code changes do not work cleanly, then the current 2015.9 (and
the documentation hotfix to follow) would become the last LAVA releases
for Ubuntu Trusty 14.04LTS.
Users of Ubuntu Trusty 14.04LTS are reminded that a new LTS is due for
release in April 2016 but that the LAVA software team are not using
Trusty any longer. Any issues with the migration from Trusty to the
new LTS are outside of our control.
Migrating an instance from Ubuntu to Debian would involve dumping and
re-importing the database - bear in mind that Trusty has postgres9.3 and
Jessie has 9.4, so care will be needed to migrate the postgresql
cluster correctly - there is documentation for that migration in the
lava-server docs. It is not recommended to attempt to migrate the OS
itself from Ubuntu to Debian (or vice-versa), a reinstall is a better
option, so make a backup and ensure that the backup works.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
After discussions at the LAVA Users Forum at Connect SFO15 [0], the LAVA
team have arranged for some changes in the mailing lists to make it easier
to announce and manage the upcoming changes from the ongoing refactoring.
https://lists.linaro.org/mailman/listinfo/lava-users - a discussion list
amongst users for users, with some admins listening in and helping out.
Users in this sense includes test writers and administrators of LAVA
instances outside Linaro. It is particularly aimed at those running LAVA in
places where Linaro is not necessarily aware of the install. I'm not asking
for LAVA instances to declare themselves, it is free software after all, it
is an offer of assistance and a request that users take advantage of the
support available to manage the implications of the refactoring for their
own use cases. Please feel free to initiate threads and discuss issues and
problems on the lava-users list.
As a reminder, for those who use LAVA but have not recently been to a
Connect, the dispatcher is being refactored and this had led to
advancements and modifications in the lava-server as well as a completely
re-written job submission format. LAVA is retaining compatibility *only*
with the Lava Test Shell Definitions (the YAML files people are currently
using) and there can be no automated way of converting existing JSON job
submissions to the new job submission format (which uses YAML to allow for
comments, amongst other improvements).
The timetable for these changes is expected to cover most of 2016. After
https://validation.linaro.org/ has migrated to the new code, the current
dispatcher code will start to be removed from the upstream in 2017.
One result of these changes is that as the Linaro production instance
migrates away from the current dispatcher codebase, there is a likelihood
of some breakage in some parts of the current dispatcher. (This is one of
the primary reasons why the dispatcher is being refactored.)
The refactoring introduces a lot of benefits, including much more robust
communication between the workers and the master, removal of configuration
on the workers so that admins only change things in one place, a lot of new
methods within the dispatcher to support new types of test and a much
cleaner, more modular, codebase for future development.
Also, the Linaro Validation Mailman List <linaro-validation(a)lists.linaro.org>
is to be renamed (keeping linaro-validation as an alias) to lava-devel for
consistency with lava-users and lava-announce. There will be a separate
email to the linaro-validation list about these changes shortly.
These new mailing lists will be included in the documentation included in
next release of the LAVA software (2015.10). Documentation of the new code
is already in the master branch of lava-server and installed via the
lava-server-doc package and the initial code for the refactoring is being
developed in parallel with the current code. This list will be used to
announce when the refactoring codebase is ready for admins to begin
migrating tests and test writers to the new code and methods - currently,
the new dispatcher is developer only. At SFO15, there was also an update on
the state of the refactoring. [1]
lava-announce will continue to be a low volume list. Reply-To has been set
to the linaro-validation mailing list (transparently migrating to
lava-devel once the linaro-validation alias is in place).
The LAVA developers would ask that everyone running LAVA would subscribe to
at least the lava-announce mailing list, to help with the migration to the
new support.
Videos will be available for most meetings at SFO15 in due course. The
slides are already available.
[0] https://sfo15.pathable.com/meetings/302656
[1] https://sfo15.pathable.com/meetings/303074
--
Neil Williams
=============
Tech Lead, LAVA software team.
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
