This is the first full release from the combined source tree.
The expected direction of the LAVA codebase is to add more devices and the
work of device integration is and has always been difficult. Merging the
codebase removes an unnecessary hurdle in integrating new devices and
smoothes the process of ensuring that existing integrations are maintained
by generating dispatcher test files directly from the server templates.
(This has been a problem in previous releases - static device configuration
files in lava_dispatcher unit tests hid problems arising from changes to
the templates in lava_scheduler_app. The unit tests are now more
representative of real test job processing.) As a validation framework, our
own unit test and CI support is a key technical component.
lava_scheduler_app and lava_dispatcher modules will remain tightly
integrated, using ZMQ to allow for physical separation. Combining the tree
also makes it easier to update the documentation each time a new device is
integrated.
The new lava-common binary package allows lava-server to be installed
without needing lava-dispatcher - this will reduce the number of
dependencies required on instances with multiple workers where there are no
devices attached to the master. The default remains to install
lava-dispatcher alongside lava-server - use the --no-install-recommends
option, as per the updated documentation.
There are changes required in Debian for this release and those will take
some time. See https://ftp-master.debian.org/new.html and
https://ftp-master.debian.org/new/lava_2018.5-1.html (It is not necessary
for every package in that queue to be processed before the LAVA upload is
processed, some packages are held back for specific reasons.) Once the LAVA
upload clears NEW, the old lava-dispatcher and lava-server source packages
will be removed from Debian.
One key advantage of this change within Debian is that all lava packages
will migrate together - this will prevent a recurrence of the recent
problem when lava-dispatcher was delayed arriving in backports but
lava-server was accepted, causing installation failures.
Both lava-server and lava-dispatcher binary packages depend on the same
version of lava-common. When upgrading, lava-common should be added
automatically. If not, use dist-upgrade or list the packages explicitly to
let apt add the new package.
e.g.
$ sudo apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
lava-common
The following packages will be upgraded:
lava-dev lava-dispatcher libprocps6 procps
Once lava-common is installed, if a master has no devices attached, admins
can choose to remove lava-dispatcher and the extra dependencies can also be
removed. *Check the list of packages to be removed on each instance* for
packages required by other tools on the master, e.g. local admin scripts
and similar.
$ sudo apt update
$ sudo apt upgrade
or to list packages explicitly:
# master with devices (may also have workers):
$ sudo apt update
$ sudo apt install lava-server lava-dispatcher
# or for workers:
$ sudo apt install lava-dispatcher
# or for master without devices:
$ sudo apt install lava-server
$ sudo apt remove lava-dispatcher
$ sudo apt-get --purge autoremove
lava changes
============
4a4bde2a1 LAVA-999 Export master and logger config
6f50e1145 LAVA-1315 add option to update existing users
ff8d9f882 Update device-types to match my lab config
809ae34ea Use InfrastructureError for uefi menu timeout
e52d83dc6 Stop the Overlay tests duplicating the base class
099f19599 LAVA-1310 separate the template unit tests
198283abb Drop Python2 xmlrpc support
13fa6bdef Fix support for scheduler.jobs.definition
a40fe8664 Fix query results export to CSV.
b18e74ca9 Create lava_common for Timeout and Exceptions
ece1a46d4 Generate device configuration from templates
e2d0f4b36 LAVA-1324 QEMU usage of KVM accelerator
59e0ab091 Extend timeouts in example test jobs
4269cc55a Merge branch 'master' into coordinator
7f49853f0 Complete fastboot sequence cannot be governed in device
dictionary.
50d69d971 Update instructions for merged codebase
e2de099f4 LAVA-1316 Instant scheduling
edd48e0b1 signals: always add health_check field
1bbaec1e2 Allow to create health-check directly
e0cddeb5e Create admin log entries when looping
2489bcbc8 Add auto-login to fastboot sequence of hi6220 device types.
13a4b5923 Drop debug line to update pipeline_reference
da933a151 Add support for optionally running using pytest-3
51c81f659 Test decompression during download streams
3d2541d00 docker: always run with --rm
1cdd275aa Fix ResetDevice comment to match what it actually does.
db2775001 Allow to sleep after the usb serial appeared
391595bcc LAVA-1309 - pre-* commands should be called with minus-lxc
d524354f7 Add an helper to list all keys of an action namespace
9bfe50d84 Expand command lists in device dictionaries
0ff09a569 LAVA-1173 - allow for matching parents of nodes
3b5f47bf7 Make device_info validate errors unique
dad039adb Allow for multiple udev rules in a single file
80ca5346a LAVA-1203 - Allow multiple callback URLs in notify action
120181cb6 ci-run should stop on first error
bed5fcb20 Allow dispatcher unit tests to generate device config
a3cc054f3 Implementation of recovery mode deploy and boot
47b184796 Print the url when validating file:// download
8d1ffad00 dispatcher: raise a TestError for invalid regexp
fb39a9da8 Factorize compatiblity code
4511d4de6 monitor: log the start message
c18752652 Fix broken unit test
84769209a Exclude Retired devices from invalid_template check
534f1a9b8 LAVA-1300 - fail if static devices cannot be found
235f17d35 Add the end time to the Recent job errors table
e18d27c5a Update templates for changes from Harston lab
fa0d9125f Update second UART example jobs
600fc7b4e ci-run: Remove some of the unused args and clean it up for jenkins
36fee805b Revert "Remove unused and obsolete requirements.txt"
2e051cb6d Remove any generated top level init.py
2060d7851 LAVA-1298 fix references to old repositories
1978274d1 Treat kernel panics as a job failure
d920b7825 Remove unused and obsolete requirements.txt
ed9039e87 LAVA-1297 - Incorporate changes to run jobs without LXC
fabd4a44f Port to python3 and fix prospector warning
e98737c55 LAVA-1286 Raise JobError if 'params' is not a dict
5af7a7f74 compression: do not use shell=True
a4e1ac09b device-types: add meson-gxbb-nanopi-k2
2e84699d8 Port classes to python3 only
6a02322a1 device-types: add meson-gxl-s905x-libretech-cc
968873081 publisher: do not duplicate logs
05d119426 Send the wait-all role reply
80a912d7e Add a ./ci-run script
02a83fed4 Fix waitall with a role support
33644d89e Port to new daemon support with systemd service
a1d9f99e3 Explicitly allow wait-all to occur before send
8b2e3feb0 Add a status check script
a4cfb1b90 LAVA-489 Python3 support
1c570a0f8 add the gitreview file for lava-coordinator
210e4b7cf bump version
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
On 24 April 2018 at 14:09, Zoran S <zoran.stojsavljevic.de(a)gmail.com> wrote:
> > Sorry, but this was only a notification of a decision already taken and
> implemented by
> > the development team. The change itself was not and is not up for
> discussion on this list.
>
> Sorry to be (brutal) honest with you and Linaro. ;-)
>
> I (sincerely) expect in The Future you/Linaro will be MUCH (much) more
> serious with some responsibilities to this list, and to your/Linaro
> Lava architecture?
>
This is a user list. It is not a development list.
I remain unconvinced that you, personally, understand the LAVA architecture
and design let alone our aims and objectives.
>
> Or... I expect out there that will be some more serious
> people/companies with some more serious testing environment. I hope
> for this to happen (competition is All Good, don't you think)! ;-)
>
> Do we have agreement here (I truly hope so)?! :-)
>
Not really.
LAVA is absolutely serious about automation validation. I disagree with all
the rest of your assumptions.
>
> Zoran Stojsavljevic
>
> On Tue, Apr 24, 2018 at 10:02 AM, Neil Williams
> <neil.williams(a)linaro.org> wrote:
> > On 24 April 2018 at 08:21, Zoran S <zoran.stojsavljevic.de(a)gmail.com>
> wrote:
> >>
> >> > Not true. The master produces the configuration which is fed to the
> >> > dispatcher - the two are very directly tied
> >> > at the level of what devices can support and how that support is used
> by
> >> > test jobs. Changes and fixes in the
> >> > device-type templates will change how the dispatcher uses the code
> >> > support to control the device.
> >>
> >> With all due respect, Sir, I would not agree/would disagree to by
> >> your/Linaro's currently presented architecture.
> >
> >
> > Sorry, but this was only a notification of a decision already taken and
> > implemented by the development team. The change itself was not and is
> not up
> > for discussion on this list.
> >
> >>
> >> On Mon, Apr 23, 2018 at 4:06 PM, Neil Williams <
> neil.williams(a)linaro.org>
> >> wrote:
> >> > On 23 April 2018 at 14:54, Zoran S <zoran.stojsavljevic.de(a)gmail.com>
> >> > wrote:
> >> >>
> >> >> > lava-server and lava-dispatcher have been merged into a single
> source
> >> >> > repository - lava
> >> >>
> >> >> If you ask me, this is the (very) questionable decision. Indeed!?
> >> >>
> >> >> lava-server is the front-end manager of the whole Lava test
> >> >> environment, while lava-dispatcher is the back-end.
> Connected/splitted
> >> >> by ZMQ protocol. And... They should be separately maintained
> >> >
> >> >
> >> > Not true. The master produces the configuration which is fed to the
> >> > dispatcher - the two are very directly tied at the level of what
> devices
> >> > can
> >> > support and how that support is used by test jobs. Changes and fixes
> in
> >> > the
> >> > device-type templates will change how the dispatcher uses the code
> >> > support
> >> > to control the device. The two have always been maintained by the same
> >> > team
> >> > and in a tightly integrated manner.
> >> >
> >> >>
> >> >> , since
> >> >> they represent (very) different things. Essentially, they do (very)
> >> >> different tasks. They are, after all, apples and oranges.
> >> >
> >> >
> >> > The bulk of the expected work from here on is Device Integrations.
> Each
> >> > new
> >> > device integration typically involves changes to the dispatcher to
> >> > support
> >> > new deployment / boot methods as well as new templates for the device
> >> > configuration. Combining the codebase allows those changes to be more
> >> > easily
> >> > verified because the output of the device configuration templates can
> be
> >> > fed
> >> > directly to the lava-dispatcher code changes in the dispatcher unit
> >> > tests.
> >> >
> >> > Previously, this has resulted in static device configuration files in
> >> > the
> >> > dispatcher unit tests and that has caused problems.
> >> >
> >> >>
> >> >>
> >> >> In other words, if you update/buy advanced/more expensive apples, you
> >> >> also force testers to update for nuthin'/buy for the loss the same
> >> >> oranges, they had before?!
> >> >
> >> >
> >> > Not true. the packages are built from a single source package. That
> does
> >> > not
> >> > mean that the binary packages need to be updated on the actual
> >> > instances.
> >> >
> >> > Up until now, lava-server has always depended on the latest version of
> >> > lava-dispatcher. That specific dependency is being removed as part of
> >> > this
> >> > change.
> >> >
> >> > Nevertheless, the principle remains that the lava-master and
> lava-slave
> >> > should be updated together so that the device configuration is in line
> >> > with
> >> > the dispatcher support. See compatibility settings in the
> documentation.
> >> >
> >> > https://lava.codehelp.co.uk/static/docs/v2/simple-admin.html#index-5
> >> >
> >> >
> >> >
> >> >>
> >> >>
> >> >> Is it the wise decision???
> >> >>
> >> >> Two cents worth lamenting/analysis,
> >> >> Zoran
> >> >> _______
> >> >>
> >> >> On Thu, Apr 19, 2018 at 3:41 PM, Neil Williams
> >> >> <neil.williams(a)linaro.org>
> >> >> wrote:
> >> >> > This is for particular note for developers as it changes the way
> that
> >> >> > reviews happen.
> >> >> >
> >> >> > If you've noticed a few reviews being abandoned, this is why.
> >> >> >
> >> >> > lava-server and lava-dispatcher have been merged into a single
> source
> >> >> > repository - lava
> >> >> >
> >> >> > lava-coordinator will follow in time, to ease the update of
> >> >> > lava-coordinator
> >> >> > to Python3.
> >> >> >
> >> >> > This will, in future, allow easier testing of device integrations
> by
> >> >> > allowing the lava_scheduler_app unit tests to be linked to the
> >> >> > lava_dispatcher unit tests and have a single review which adds both
> >> >> > sides of
> >> >> > the device support.
> >> >> >
> >> >> > There will be a lot of testing, as normal, so staging will be
> moving
> >> >> > to
> >> >> > packages built from the new source repository tree.
> >> >> >
> >> >> > The old lava-server.git and lava-dispatcher,git repositories will
> >> >> > become
> >> >> > read-only and will get no further code changes. All changes will be
> >> >> > done
> >> >> > in
> >> >> > lava.git
> >> >> >
> >> >> > https://git.linaro.org/lava/lava.git/
> >> >> >
> >> >> > The documentation will be updated over the next few days.
> >> >> >
> >> >> > --
> >> >> >
> >> >> > Neil Williams
> >> >> > =============
> >> >> > neil.williams(a)linaro.org
> >> >> > http://www.linux.codehelp.co.uk/
> >> >> >
> >> >> > _______________________________________________
> >> >> > Lava-announce mailing list
> >> >> > Lava-announce(a)lists.linaro.org
> >> >> > https://lists.linaro.org/mailman/listinfo/lava-announce
> >> >> >
> >> >> _______________________________________________
> >> >> Lava-users mailing list
> >> >> Lava-users(a)lists.linaro.org
> >> >> https://lists.linaro.org/mailman/listinfo/lava-users
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Neil Williams
> >> > =============
> >> > neil.williams(a)linaro.org
> >> > http://www.linux.codehelp.co.uk/
> >
> >
> >
> >
> > --
> >
> > Neil Williams
> > =============
> > neil.williams(a)linaro.org
> > http://www.linux.codehelp.co.uk/
>
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
2018.4.post2 hot fix is now available.
This is the first release built from the combined source tree but the
actual code changes are restricted to fixes for decompression of downloaded
files, specifically .zip and .xz handling.
Python3 only and Stretch only.
Now available from the LAVA repositories, production-repo.
https://staging.validation.linaro.org/static/docs/v2/installing_on_debian.h…
Changes
=======
fe70ac6ca Update version.py for repos with no tags
1f7c5b5f1 Fix typo in 86362f and add debug
11c5ab57e Do not attempt to unzip on the fly
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
On 24 April 2018 at 08:21, Zoran S <zoran.stojsavljevic.de(a)gmail.com> wrote:
> > Not true. The master produces the configuration which is fed to the
> dispatcher - the two are very directly tied
> > at the level of what devices can support and how that support is used by
> test jobs. Changes and fixes in the
> > device-type templates will change how the dispatcher uses the code
> support to control the device.
>
> With all due respect, Sir, I would not agree/would disagree to by
> your/Linaro's currently presented architecture.
Sorry, but this was only a notification of a decision already taken and
implemented by the development team. The change itself was not and is not
up for discussion on this list.
> On Mon, Apr 23, 2018 at 4:06 PM, Neil Williams <neil.williams(a)linaro.org>
> wrote:
> > On 23 April 2018 at 14:54, Zoran S <zoran.stojsavljevic.de(a)gmail.com>
> wrote:
> >>
> >> > lava-server and lava-dispatcher have been merged into a single source
> >> > repository - lava
> >>
> >> If you ask me, this is the (very) questionable decision. Indeed!?
> >>
> >> lava-server is the front-end manager of the whole Lava test
> >> environment, while lava-dispatcher is the back-end. Connected/splitted
> >> by ZMQ protocol. And... They should be separately maintained
> >
> >
> > Not true. The master produces the configuration which is fed to the
> > dispatcher - the two are very directly tied at the level of what devices
> can
> > support and how that support is used by test jobs. Changes and fixes in
> the
> > device-type templates will change how the dispatcher uses the code
> support
> > to control the device. The two have always been maintained by the same
> team
> > and in a tightly integrated manner.
> >
> >>
> >> , since
> >> they represent (very) different things. Essentially, they do (very)
> >> different tasks. They are, after all, apples and oranges.
> >
> >
> > The bulk of the expected work from here on is Device Integrations. Each
> new
> > device integration typically involves changes to the dispatcher to
> support
> > new deployment / boot methods as well as new templates for the device
> > configuration. Combining the codebase allows those changes to be more
> easily
> > verified because the output of the device configuration templates can be
> fed
> > directly to the lava-dispatcher code changes in the dispatcher unit
> tests.
> >
> > Previously, this has resulted in static device configuration files in the
> > dispatcher unit tests and that has caused problems.
> >
> >>
> >>
> >> In other words, if you update/buy advanced/more expensive apples, you
> >> also force testers to update for nuthin'/buy for the loss the same
> >> oranges, they had before?!
> >
> >
> > Not true. the packages are built from a single source package. That does
> not
> > mean that the binary packages need to be updated on the actual instances.
> >
> > Up until now, lava-server has always depended on the latest version of
> > lava-dispatcher. That specific dependency is being removed as part of
> this
> > change.
> >
> > Nevertheless, the principle remains that the lava-master and lava-slave
> > should be updated together so that the device configuration is in line
> with
> > the dispatcher support. See compatibility settings in the documentation.
> >
> > https://lava.codehelp.co.uk/static/docs/v2/simple-admin.html#index-5
> >
> >
> >
> >>
> >>
> >> Is it the wise decision???
> >>
> >> Two cents worth lamenting/analysis,
> >> Zoran
> >> _______
> >>
> >> On Thu, Apr 19, 2018 at 3:41 PM, Neil Williams <
> neil.williams(a)linaro.org>
> >> wrote:
> >> > This is for particular note for developers as it changes the way that
> >> > reviews happen.
> >> >
> >> > If you've noticed a few reviews being abandoned, this is why.
> >> >
> >> > lava-server and lava-dispatcher have been merged into a single source
> >> > repository - lava
> >> >
> >> > lava-coordinator will follow in time, to ease the update of
> >> > lava-coordinator
> >> > to Python3.
> >> >
> >> > This will, in future, allow easier testing of device integrations by
> >> > allowing the lava_scheduler_app unit tests to be linked to the
> >> > lava_dispatcher unit tests and have a single review which adds both
> >> > sides of
> >> > the device support.
> >> >
> >> > There will be a lot of testing, as normal, so staging will be moving
> to
> >> > packages built from the new source repository tree.
> >> >
> >> > The old lava-server.git and lava-dispatcher,git repositories will
> become
> >> > read-only and will get no further code changes. All changes will be
> done
> >> > in
> >> > lava.git
> >> >
> >> > https://git.linaro.org/lava/lava.git/
> >> >
> >> > The documentation will be updated over the next few days.
> >> >
> >> > --
> >> >
> >> > Neil Williams
> >> > =============
> >> > neil.williams(a)linaro.org
> >> > http://www.linux.codehelp.co.uk/
> >> >
> >> > _______________________________________________
> >> > Lava-announce mailing list
> >> > Lava-announce(a)lists.linaro.org
> >> > https://lists.linaro.org/mailman/listinfo/lava-announce
> >> >
> >> _______________________________________________
> >> Lava-users mailing list
> >> Lava-users(a)lists.linaro.org
> >> https://lists.linaro.org/mailman/listinfo/lava-users
> >
> >
> >
> >
> > --
> >
> > Neil Williams
> > =============
> > neil.williams(a)linaro.org
> > http://www.linux.codehelp.co.uk/
>
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
On 23 April 2018 at 14:54, Zoran S <zoran.stojsavljevic.de(a)gmail.com> wrote:
> > lava-server and lava-dispatcher have been merged into a single source
> repository - lava
>
> If you ask me, this is the (very) questionable decision. Indeed!?
>
> lava-server is the front-end manager of the whole Lava test
> environment, while lava-dispatcher is the back-end. Connected/splitted
> by ZMQ protocol. And... They should be separately maintained
Not true. The master produces the configuration which is fed to the
dispatcher - the two are very directly tied at the level of what devices
can support and how that support is used by test jobs. Changes and fixes in
the device-type templates will change how the dispatcher uses the code
support to control the device. The two have always been maintained by the
same team and in a tightly integrated manner.
> , since
> they represent (very) different things. Essentially, they do (very)
> different tasks. They are, after all, apples and oranges.
>
The bulk of the expected work from here on is Device Integrations. Each new
device integration typically involves changes to the dispatcher to support
new deployment / boot methods as well as new templates for the device
configuration. Combining the codebase allows those changes to be more
easily verified because the output of the device configuration templates
can be fed directly to the lava-dispatcher code changes in the dispatcher
unit tests.
Previously, this has resulted in static device configuration files in the
dispatcher unit tests and that has caused problems.
>
> In other words, if you update/buy advanced/more expensive apples, you
> also force testers to update for nuthin'/buy for the loss the same
> oranges, they had before?!
>
Not true. the packages are built from a single source package. That does
not mean that the binary packages need to be updated on the actual
instances.
Up until now, lava-server has always depended on the latest version of
lava-dispatcher. That specific dependency is being removed as part of this
change.
Nevertheless, the principle remains that the lava-master and lava-slave
should be updated together so that the device configuration is in line with
the dispatcher support. See compatibility settings in the documentation.
https://lava.codehelp.co.uk/static/docs/v2/simple-admin.html#index-5
>
> Is it the wise decision???
>
> Two cents worth lamenting/analysis,
> Zoran
> _______
>
> On Thu, Apr 19, 2018 at 3:41 PM, Neil Williams <neil.williams(a)linaro.org>
> wrote:
> > This is for particular note for developers as it changes the way that
> > reviews happen.
> >
> > If you've noticed a few reviews being abandoned, this is why.
> >
> > lava-server and lava-dispatcher have been merged into a single source
> > repository - lava
> >
> > lava-coordinator will follow in time, to ease the update of
> lava-coordinator
> > to Python3.
> >
> > This will, in future, allow easier testing of device integrations by
> > allowing the lava_scheduler_app unit tests to be linked to the
> > lava_dispatcher unit tests and have a single review which adds both
> sides of
> > the device support.
> >
> > There will be a lot of testing, as normal, so staging will be moving to
> > packages built from the new source repository tree.
> >
> > The old lava-server.git and lava-dispatcher,git repositories will become
> > read-only and will get no further code changes. All changes will be done
> in
> > lava.git
> >
> > https://git.linaro.org/lava/lava.git/
> >
> > The documentation will be updated over the next few days.
> >
> > --
> >
> > Neil Williams
> > =============
> > neil.williams(a)linaro.org
> > http://www.linux.codehelp.co.uk/
> >
> > _______________________________________________
> > Lava-announce mailing list
> > Lava-announce(a)lists.linaro.org
> > https://lists.linaro.org/mailman/listinfo/lava-announce
> >
> _______________________________________________
> Lava-users mailing list
> Lava-users(a)lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/lava-users
>
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
This is for particular note for developers as it changes the way that
reviews happen.
If you've noticed a few reviews being abandoned, this is why.
lava-server and lava-dispatcher have been merged into a single source
repository - lava
lava-coordinator will follow in time, to ease the update of
lava-coordinator to Python3.
This will, in future, allow easier testing of device integrations by
allowing the lava_scheduler_app unit tests to be linked to the
lava_dispatcher unit tests and have a single review which adds both sides
of the device support.
There will be a lot of testing, as normal, so staging will be moving to
packages built from the new source repository tree.
The old lava-server.git and lava-dispatcher,git repositories will become
read-only and will get no further code changes. All changes will be done in
lava.git
https://git.linaro.org/lava/lava.git/
The documentation will be updated over the next few days.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
The 2018.4 release has been uploaded to Debian unstable. It should also be
possible to upload to Stretch-backports in about a week from now, depending
on internal processes in Debian.
The release is also available from the LAVA repositories - production-repo
Dropping Python2
================
2018.4 is the first release exclusively using Python3. Upgrading from
releases prior to 2018.2 will bring in a range of Python3 packages. After
installation of 2018.4, a number of Python2 packages are available for
removal, if administrators desire.
See also: https://lists.linaro.org/pipermail/lava-announce/
2018-April/000050.html
lava-tool still uses Python2 - we are working on lavacli as a Python3
replacement. lavacli has also been updated to 0.8-1 in Debian.
lava-server
===========
f60a5328 callback: fix crash in python3
e627f8ec Allow skipping one unit test if kvm not enabled
0208402f job: make the link to the bottom more visible
dd26e4fb Fix default BRANDING_BUG_URL
b2cc847b Adjust bootm addresses for panda
f25548b5 Make snow u-boot prompt more precise
6a209317 Misc beaglebone black fixes
1481ac98 Add note for bug in LXC templates on Stretch
4ed6a2f5 api: fix a crash when a test case is not unique
53c57de9 Fix crash when test case metadata is empty
99c32f4d Fix crash when action_metadata is None
b286b6bd Fix missing import
b708b038 Test user_command and flasher templates
c316e516 Update more scripts to use Python3
4be0dd49 fix sphinx warning in docs
458f470e Improve "lava-server manage check" coverage
30efadeb LAVA-1274 fix javascript exception
31764a52 job: only update if failure_comment isn't empty
24fe81e6 LAVA-1276 - Search functionality broken on staging job list page
938a90d8 Update documentation for Python3 change
d4f64157 Fix deprecation warning from Jinja2 with Python3
1a5c55e8 Add hi6220-hikey-r2 device-type template
13758edc Update development notes for Python3
8f0d5a94 Pass LXC/fastboot parameters to all jobs in split_multinode_yaml
f3d7a8ac Fix API breakage from 2d37b3
2658935c Fix Python2 syntax when creating secret key.
65a30238 Fix Device.CONFIG_PATH for individual unit tests
c5fbfa51 Use suppress from contextlib
60b50af3 Remove unused shebang
c9ff6e9f Fix shebang for python3
494ae3af Fix wrong import
33c5f72f LAVA-1269 - Store checksum of job.original_definition in metadata
f8124a8c Import infrastructure_error from the right module.
60ecef88 Add failure comment and error type to test job notifications.
4e313b45 Fix logic for template check warning
ab65a859 Fix the logic in e1f44097
b0c549bb Fix unicode error in migrations
ec01c5bf Replace "!!python/unicode" in job definitions.
f4386c08 LAVA-1147 Support more than 3 priorities
c169c287 Remove exclusive flag from device dict
d7cba2d1 LAVA-1250 Allow admin to finish a TestJob
3b54f8d2 Preserve comments in job definition display
66cd46bd LAVA-1167 document the 'history' feature
8d835d78 Python3 changes needed for lava-server
73fa8d2e device-types: add sharkl2
48f63bf9 Fix missing conditions variable in custom query manager.
bdb07a21 Fix pep8 errors
70614729 Fix crash when device dict is invalid
b00fde96 Remove jessie specific code
fa0418b7 Improve the Admin query view
7d8d94de Skip refresh of archived queries
926ba5af Use the TestCase job error message as failure comment
e1f44097 Fix invalid_template check
ea3d99ca LAVA-1263 separate vexpress bootloader_prompts
0ef60b0e Add admin link to device pages for superuser
bba64496 Cleanup some doc error messages
91c03b30 Fix multinode sub job definition display.
48bb281e Job resubmission shows Internal Server Error
on incorrect resubmission.
7b2201b5 Cancel subjobs from admin cancel action
aac25876 Fix a grub error message so python can compile it into an RE
3555256f device-types: add sun8i-h2-plus-orangepi-r1
1ae6dda0 device-types: add sun8i-h2-plus-orangepi-zero
f5e95c3a Save the error when the job fails to start
1fbd68fa allow load_devicetype_template to return raw yaml
2667881c dt: alert when the template is invalid or missing
75cf05ee Move infrastructure_error into test.utils
5819a2cd Add banana pi zero device-type config
d8f605c0 Add another possible failure message to grub
e5aca37a LAVA-1129 save job full configuration before start
f54c2acf Make some after installation debug doc more visible.
4bb83a63 Fix 500 when device-type template does not exist
e1371fef juno: allow override of bootloader_prompt
b33a3165 Make the code python3 compatible after 0d1e0a07
d73687c6 LAVA-1253 - CSV export headers are incorrect
14aa6477 Force device hc for ConfigurationError and LAVABug
02011867 Use base-fastboot jinja2 template for mediatek-8173 device type.
606ad8b0 LAVA-1240 - LAVA: add test job name to email notification subject
414edc4d Add the overlay deploy method to all uboot devices
f1c8d9c7 Add the flasher configuration to all uboot devices
fec9dfac settings: simplify and allow setting any variables
f2f6844d LAVA-1247 - Results API make_custom_query not working correctly
1a27b586 Remove unused variables from instance.conf
de0834d6 settings: remove most variables from settings.conf
24646594 settings: remove appname
3f69dd81 Fix lava-server manage addldapuser --username USER
5aebdfce admin: fix view_siet link when using MOUNT_POINT
f784db06 admin: use dynamic urls instead of static ones
0d1e0a07 Fix test for extra results metadata in callback data
356ba428 Move standard ARMMP tests to stretch
227caa82 LAVA-1249 address issues with large pexpect buffers
4bdb5d17 Allow time to drain capacitance
717a3387 Up power off timeout for juno
7a1260d4 Fix bootloader prompt for juno uboot
ca334c5c job.validate does not accept any parameter anymore
fa0855de Remove v1 leftover
5287d13d Remove unused output_dir
e9fc34f0 device-types: rk3399-puma-haikou: align booti_kernel_addr
33c8579b Add imx7s-warp device-type
416fb3e5 Do not crash when re-adding the same device (or dt)
18318d27 Add admin actions to update worker health
f57d683d master|logs: automatically reload certificates
390c8e2b lava-master: fix return values
2d37b341 Remove is_pipeline
45d5767a lava-master: record error message as failure_comment
74713edb Use STATIC_URL and get_absolute_url when needed
60283360 Remove TestJob._results_link
f9a36ee1 LAVA-1180 Refactor device-type page
3456b9ea LAVA-1128 Use bulk_create to create TestCases
cfcc0fc8 Move postgresql timeout to wsgi
4c61a59c LAVA-777 lava-slave is now restartable
806ac08b job api: allow filtering by state/health
fb0b700e device-types api: allow get/set health-checks
6eadfc62 List Infrastructure, Bug and Configuration errors
lava-dispatcher
===============
4dd73b76a Fix wrong infra error when compressing with xz
50a652a54 Add missing import and fix prospector warnings
817fe007d LAVA-1234 - Man page for lava-lxc-mocker
691041a1 compress: check that the tool does exists
84df8c5d LAVA-1264 improve handling of cancellation
02590e0b LAVA-1282 Only kill lava-slave
1cbebc39 fix typo in manpage
67a19da1 Add template testing for user-command and flasher
40a9a97b Do not crash when command output is not utf-8
73cd3623 LAVA-1266 - LAVA doesn't fail jobs that fail
to flash partitions
7340ad00 BootloaderCommandOverlay fix validate logic
99526367 Log str representation of exception objects
957cc171 LAVA-1278 use external decompression for .xz
ee0921ab LAVA-648 Slave needs to check DNS
896936c1 Update modprobe.d support (Closes: #888681)
67f49224 Extend installer error prompt list
43a29e4c Ensure QEMU iso method uses configured TFTP dir
e88d3cb4 Python3 update for QEMU installer support
a05f307d Decode returned log string to utf-8.
52babb39 LAVA-1274 do not log a boolean
15ba29b9 Fix shebang for python3
4c960c49 test: fix file descriptor leaks
a6fdafd9 Remove python2 specific code
3e7891d1 Drop Python2 support
8a7d655a Revert "shell: wait for prompt between each command"
d5e15af2 Fix lava-os-build output
303b3871 Remove deprecated lava-test-run-attach usage
8cb5f120 Remove leftover from v1 in android helpers
0e304335 shell: wait for prompt between each command
b67fb420 Fix raise format
351176ab Remove stale udev rules
8d6c5ec7 Python3 fixes
e8a757df LAVA-1167 allow removal of the .git directory
54ad1fbe Remove jessie specific code
f4784431 Remove v1 signal handler
bb910afd Fixup connection.prompt_str assumption
26897c46 LAVA-1257 do not crash when STARTRUN is missing
bd6c2526 LAVA-1256 raise a JobError when url is missing
0e255c3b Raise an Infra Error when an exe is not usable
294a2322 Fix URLs referred in unittests to point to permanent location.
a1474089 LAVA-1252 do not crash when device conf is missing
5179d1de Device: factorize code
c5645bc6 Remove power_on command as it's never used
bf57d6fe Fastboot boot method should have a sequence associated with it.
eab74bae Catch errors with lava_test_results_dir early
b4ac1dbe Use the permanent location for WaRP7 unit test
a41f73ee Remove empty class functions
30b84f20 Fix format string when raising
1854a1dd Do not print pipeline description to stdout
0512e6ce Remove unused output_dir
e5043cb3 run_command: split the output at the newline char
cadd2123 Add a new deploy method called "flasher"
00437106 Use job tmp directory for the compress overlay
1158c5e7 Add a deploy method that only create the overlay
eeac3e67 docker: only mount the overlay when available
6916519d Make the helpers faster
d8b205ee Specify the exception to raise when timing out
ad106451 LAVA-1146 lava-slave removes stale resources
67b18778 LAVA-777 make lava-slave restartable
b61a58e5 Refactor lava-slave code
cdd2ef54 Adding deployment action for MPS devices
b8e7dc13 xnbd: fix killing of old xnbd-server process
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
The master branches of lava-server and lava-dispatcher have now moved to
Python3 exclusively.
This means that all developer builds will only support Python3 and that
Debian Jessie is no longer supported for LAVA software.
It is recommended to use the staging-repo as there are a lot of dependency
changes to add Python3 dependencies on machines which have not used Python3
before. apt will make the job much easier than trying to install developer
packages and then fixing the dependencies manually.
Even if you have 2018.2 installed and have Python2 and Python3 support
available, there are still some new dependencies to move the runtime
support to Python3.
The following NEW packages will be installed:
gunicorn3 python3-dateutil python3-gunicorn
The documentation has been updated:
https://staging.validation.linaro.org/static/docs/v2/installing_on_debian.h…
Once an instance has been upgraded, apt will typically warn about a range
of Python2 packages which are no longer required. These can be removed
using the command:
$ sudo apt-get --purge autoremove
Check the list of packages carefully!
One example list could be:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
python-django-auth-ldap* python-django-restricted-resource*
python-django-tables2* python-ldap* python-lzma* python-markdown*
python-netifaces* python-pyudev* python-simplejson* python-voluptuous*
0 upgraded, 0 newly installed, 10 to remove and 1 not upgraded.
After this operation, 1,855 kB disk space will be freed.
Do you want to continue? [Y/n]
apt is cautious about listing packages for autoremoval, so you may well
find Python2 packages still installed. This is normal.
This starts a new round of functional testing in preparation for the next
LAVA software release. That release will be Python3 only and it will not be
built for Jessie.
Debian uploads from here on will only be to unstable and stretch-backports.
LAVA repositories will only carry updated packages for stretch-backports.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
See
https://lists.linaro.org/pipermail/lava-announce/2018-January/000046.html
The time has come to drop regular builds for jessie-backports. If you rely
on these, you will have found that the builds of the master branch broke
for jessie-backports some time ago (whilst the team were at a conference).
So, in the absence of requests to fix the build and in the light of the
existing plan to drop Jessie support in Q2 2018 (i.e. the next production
release) there seems to be no reason to retain the build.
This also means that building from the master branch for jessie is also
broken - this is not going to be fixed.
This opens the door for testing of lava-server using Python3 exclusively,
dropping Python2 support. Depending on how that testing proceeds, the next
production release could be Python3 only.
Summary
=======
* jessie-backports support in the staging-repo has ended
* developer builds of the master branch no longer need to build on Jessie
(and are currently broken)
* there will be no further uploads of lava-server to jessie-backports or
jessie-backports-sloppy in Debian.
* support for Stretch continues unaffected
* when the next production release is made, a backport to stretch-backports
in Debian is likely to be uploaded.
* There will be no further builds for Jessie for the production-repo or
jessie-backports in Debian.
* testing of full Python3 support can now begin
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Python3
=======
2018.2 is built with Python2 and Python3 support but only Python2 remains
is available for runtime (until Jessie support is removed). This results in
the built packages for Debian unstable and stretch-backports now bringing
in Python3 versions of the existing Python2 packages like python3-django.
The unit tests are routinely run using Python2 and Python3.
The packages available from the LAVA repositories reflect the same:
jessie-backports 2018.2-1+jessie - Python2 only
stretch-backports 2018.2-1+stretch - Python2 and Python3, using Python2 at
runtime.
Bootloader support changes
==========================
25859894 Better detect errors in the bootloader - this adds support to
distinguish between a bootloader failure and a kernel failure to detect
problems when the bootloader tries to start the kernel. This has an
important effect on how some test jobs run - see Quiet Kernels below. The
parallel change (7a2b3a68 Change the flow of bootloader commands so they
are executed individually) supports detecting failures to download
artifacts as distinct from failures to execute once downloaded.
Bootloader action optimisations
===============================
To support the better error detection, several bootloader actions have been
optimised. This means that different actions may be used, changing the
names you may be using for timeouts. e.g.
https://staging.validation.linaro.org/scheduler/job/209814/definition#defli…
The timeout name was u-boot-interrupt - now it is bootloader-interrupt.
The UI shows you which actions have been assembled into the pipeline.
e.g.:
bootloader-interrupt: Wait for prompt Hit any key to stop autoboot (timeout
00:02:00)
Quiet Kernels
=============
The bootloader support changes wait for an indication that the bootloader
has completed and that the kernel has started, by watching for a
kernel_start_message. If your kernel is configured to be quiet, then each
test job using that kernel *must* clear the kernel_start_message:
context:
kernel_start_message: ''
In most cases, the test job should not use 'quiet' as this hides important
debugging information from the kernel boot process.
LXC mocker
==========
lava-dispatcher now builds a second binary package which is intended to
support the use of Docker without needing to change the test job submission
which did not use Docker. e.g. submissions to LKFT use LXC and there are
issues trying to use LXC inside Docker. To make it simpler to reproduce
test jobs when debugging regressions, the lava-lxc-mocker can be installed
in Docker instead of lxc. Commands which would then have been run in the
LXC will instead be executed inside Docker directly.
CAUTION: do not install lava-lxc-mocker *except* inside other
containerisation like Docker. When using lava-lxc-mocker, create a
dedicated Docker image which handles only one DUT so that tools like
fastboot and adb which are constrained inside the LXC do not interfere with
other test jobs and devices.
If you have been using the developer build scripts and you are using 'sudo
debi' then you will need to change to using 'sudo dpkg -i' and listing the
lava-dispatcher package instead of the .changes file.
Release cadence
===============
Now that the roadmap to remove V1 is complete, the need for monthly
releases has diminished. As part of a range of other plans aimed at making
LAVA more flexible, the monthly release cadence has been dropped. We are
looking at a range of options, including wider functional testing of the
master branch, to support releases on a rough 3 month cadence but without
the added pressure of expected release dates. This will become a lot easier
once support for Jessie is dropped and the build matrix becomes simpler.
Changes in this release:
lava-server
===========
9ef41db7 Fix missing failure comment
206e5c65 manage jobs: allow to force fail a job
42a3bc31 Save TestCases even when metadata is too large
5efb92c7 lava-logs: look for lava.job results in all cases
19937074 Use the named urls instead of static urls
e60afcbc Ensure uboot_extra_error_message is usable
843a49ed Use reverse() instead of a static url
7583bb09 device-types: add meson-gxbb-p200
40cbffc8 breadcrumb: keep the last element clickable
2d5f34d8 Update docs examples metadata
5a5548ad device-types: fix broken conditional in base-depthcharge
d663a9dc device-types: add rk3399-gru-kevin
21ac9eb2 LAVA-1230 include lava-logs in encryption docs
d57e9131 device-types: rpi3 32-bit: allow custom kernel cmdline
32312dc7 device-types: rpi3 32-bit: allow extra_kernel_args append
afa28d5a device-types: rpi-3 32-bit: fixup DT load address
0c9e227b Add missing change to the X15 template
968a00dd Fix fdt load command wrapping.
156f1e7b fix quoting of serial numbers
57610c06 lava-logs: simplify log format
5301ef66 breadcrumb: the last element should not be a link
08547de9 Fix api pages titles and breadcrumb
fc03f62b LAVA-1083 - Bug links can be managed only on list pages
35146e4c Remove distinct ON because it can't be used correctly with order
by.
85f6bd67 Test shell helper support with base-fastboot
b86981ad master: do not send START to offline workers
b6cf83f8 admin: fix group cancelation
8890d270 Drop auto-login from hikey 620 template
5642e8cd Allow cancelation in Scheduling and Scheduled too.
bbf6c5bb LAVA-1220 - Update API calls and docs for scheduler changes.
742433b0 Drop invalid boot_message for hikey 6220
441919d7 Skip kernel-start-message for dynamic connections
8f4ecdcb Add a parameter to u-boot to override interrupting
8c093e7f Reorganise fastboot for multiple OS support
abb9fa5d Adding device-type config for MPS devices
240ffeda Add rk3399-puma-haikou device type and health check
bd0c7da7 TestCase: fix crash when accessing old test cases
4e22927e Set Unknown as the default option for changing health
078b6499 api: return the right http error code
e6b3ffdd Fix doc Unexpected indentation error.
b523e20f Fix the filename in header to the actual filename.
acf537fb Improve device tables
a0a87385 LAVA-1225 catch errors when creating health checks
ea8d5a72 Do not show Idle state when device is unavailable
dd4f25bd Allow custom tftp commands for loadfdt in U-Boot
a074403a LAVA-1219 allow unicode for job.original metadata
58d23f23 Python3 branch now merged into master
717c177c fix missing quote mark in docs
dcb8bd5a LAVA-1224 support for Xilinx-ZCU102 ZYNQ
fec11287 Improve python2 and 3 support
756d2e6a migrate-job-output: fix inverted logic for dry-run
fbeaa57f Add NotificationRecipient in the admin site
135654d6 Fix online devices listing
7007fda8 Fix change in 70cb792c7 to show extra_source
025bab80 LAVA-1221 change mouse pointer whenever needed
b28e4c7b LAVA-1222 use verbose names instead of constants
a281a6a7 Extend timeout to cover when dispatcher is busy
3985a0e1 Update development docs for Python3 plans
71374e3a migrate-job-output add --slow and --dry-run
9ce62f2d LAVA-1213 - move developer builds to Python3
81fbb4c4 Charts x-axis attribute should refer to job metadata
7af37d79 LAVA-1164 relicence some lava-server files
14604fb5 LAVA-1211 Document appending to kernel command line
e8eb6ec0 Add admin actions to update device health
b16e197b LAVA-1107 fix "device-types add" documentation
18be0e5b Set device health to unknown after an infra error
54ae8263 LAVA-950 use a ROUTER socket to connect to master
54ed49c2 Fix crash in lava.scheduler.jobs.logs
7a2b3a68 Change the flow of bootloader commands so they are executed
individually
lava-dispatcher
===============
7b9c3e84 Allow some deployments to not use deployment_data
d5dc6418 ssh: fix comparison
ebbd717b Remove spurious conditional in autologin
b3e4d794 LAVA-1232 - LAVA LXC Mocker
4d445841 Fixes for X15 fastboot_uboot
fd6dbec5 actions/deploy/lxc: lxc-apt-install action add non-interactive mode
1621ea5b LAVA-1202 - multiple boot + test section using transfer_overlay
fails
84d78a08 LAVA-1229 - read-feedback: failed on DUTs without telnet or ssh
0ae6815e Ensure RetryAction has protocol support.
6b96c961 LAVA-1227 - Handle cancellation within LXC
ce1421a3 Update ZCU102 device file for bootloader changes
d53e452c Fix login support for second UART
a0104eb9 Remove unused pipeline reference file.
a9f22a9c Allow for empty values in boot methods
00addc73 Fix lookup for grubmenuselector interrupt prompt
782e69d4 Allow skipping of kernel_start_message
52ac66fa Make waiting for a bootloader prompt optional
41f7d58d Not all boot methods have parameters
8b0d03cd Remove deprecated actions in boot sequence.
a3fb0095 Reorganize for hikey 960 AOSP support
7d78da6e Fix usage of FastbootReboot
161c1660 LAVA-1167 - Improve disk usage of test overlay tarball creation
7306a5c6 download: record the http error code when failing
fd1f7013 Pass USB devices to LXC via ID_FS_LABEL
8bae0785 raise an infra error when docker is not installed
414861e1 LAVA-1138 - panic not recorded by auto-login-action
548a6bb1 LAVA-1224 - support for Xilinx ZCU102
abb9869f Unit tests for 960 AOSP support
0a2f84ce Fix man pages
b15eb55e LAVA-950 use a ROUTER socket to connect to master
7a936509 Fix logrotation by using WatchedFileHandler
8f23430b Fix prospector warnings
25859894 Better detect errors in the bootloader
324615d7 LAVA-1165 - skip reading feedback on closed connection
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
