This is used to generate measurements by interacting with devices that are
not reachable as a node, and avoid target-specific support.
Each device.conf file can define a host command or script used when entering
a lava_command_run and leaving a lava_command_run, In the style of the PDU
power cycling commands.
host_hook_enter_command = <enter_hook> (command line to pass to host)
host_hook_exit_command = <exit_hook>
Hooks calling abi:
ENTRER: arg0 = <enter_hook> arg1 = "output-file-fullpath" &
EXIT: arg0 = <exit_hook> arg1 = "output-file-fullpath" &
The MO is close to lava_test_shell:
the LAVA_SIGNAL_TESTCASE search pattern will be applied to the output of the exit
hook and if a measurement is found, a test_case will be added to the bundle
stream of the job. Any file found in the logdir will be attached, starting with
stdout.log. If a {output-file-fullpath}.minetype is found, its content is used
as 'mime_type' for the attached file (as with lava_shell_test)
example, see Job http://lava.baylibre.com:10080/dashboard/streams/anonymous/lab-health/bundl… as an example of unit-test
power statistics record and display as test_cases.
Signed-off-by: Marc Titinger <mtitinger(a)baylibre.com>
---
v2: expect LAVA_SIGNAL_TESTCASE patterns like with lava_test_shell
---
lava_dispatcher/actions/lava_command.py | 46 +++++++++++++++++++++++++++++++--
lava_dispatcher/config.py | 5 ++++
2 files changed, 49 insertions(+), 2 deletions(-)
diff --git a/lava_dispatcher/actions/lava_command.py b/lava_dispatcher/actions/lava_command.py
index 7bf787a..0748b51 100644
--- a/lava_dispatcher/actions/lava_command.py
+++ b/lava_dispatcher/actions/lava_command.py
@@ -33,6 +33,8 @@ from lava_dispatcher.test_data import create_attachment
from lava_dispatcher.utils import read_content
from datetime import datetime
+from lava_dispatcher.actions import lava_test_shell
+from lava_dispatcher.lava_test_shell import parse_testcase_result
class cmd_lava_command_run(BaseAction):
@@ -54,16 +56,30 @@ class cmd_lava_command_run(BaseAction):
_parser = None
_fixupdict = {}
_results_from_log_file = []
+ _cur_record = None
+ _record_index = 0
+ _uuid = None
def run(self, commands, parser=None, iterations=1, fixupdict=None, timeout=-1):
target = self.client.target_device
+ context = self.context
log_dir = tempfile.mkdtemp(dir=target.scratch_dir)
self._logfile = os.path.join(log_dir, 'stdout.log')
+ self._uuid = str(uuid4())
if parser is not None:
self._parser = parser
if fixupdict is not None:
self._fixupdict = fixupdict
logging.info("lava_command logfile: %s" % self._logfile)
+
+ #if there is a host-side hook to call:
+ host_enter_hook = context.device_config.host_hook_enter_command
+ if host_enter_hook:
+ self._cur_record = os.path.join(log_dir, self._uuid.split('-')[0])
+ host_enter_hook = host_enter_hook.rstrip('&') + " " + self._cur_record + " &"
+ logging.warning('Running enter hook on host %s' % host_enter_hook)
+ context.run_command(host_enter_hook)
+
with self.client.tester_session() as session:
for count in range(iterations):
logging.info("Executing lava_command_run iteration: %s" % count)
@@ -81,6 +97,27 @@ class cmd_lava_command_run(BaseAction):
self._results_from_log_file.append(res)
logging.error(e)
+ #if there is a host-side hook to call:
+ host_exit_hook = context.device_config.host_hook_exit_command
+ if host_exit_hook:
+ host_exit_hook = host_exit_hook.rstrip('&') + " " + self._cur_record + " &"
+ logging.warning('Running EXIT hook on dispatcher host %s' % host_exit_hook)
+ output = context.run_command_get_output(host_exit_hook)
+
+ # See https://github.com/BayLibre/iio-capture as an instance of an
+ # app that will produce compatible output when called from the host
+ # Hook.
+ test_pattern = r"<LAVA_SIGNAL_TESTCASE TEST_CASE_ID=(?P<test_case_id>.*)\s+"\
+ "RESULT=(?P<result>(PASS|pass|FAIL|fail|SKIP|skip|UNKNOWN|unknown))\s+"\
+ "UNITS=(?P<units>.*)\s+MEASUREMENT=(?P<measurement>.*)>"
+ test_case_pattern = re.compile(test_pattern)
+
+ for line in output.split(os.linesep):
+ match = test_case_pattern.match(line.strip())
+ if match:
+ res = parse_testcase_result(match.groupdict())
+ self._results_from_log_file.append(res)
+
bundle = self._get_bundle()
self._write_results_bundle(bundle)
@@ -126,12 +163,17 @@ class cmd_lava_command_run(BaseAction):
def _get_test_runs(self):
now = datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%SZ')
- attachment = [create_attachment(os.path.basename(self._logfile), read_content(self._logfile))]
+ attachment_dir = os.path.dirname(self._logfile)
+ attachment = lava_test_shell._attachments_from_dir(os.path.dirname(self._logfile))
+ # fixup default mimetypes, for stdout.log mainly
+ for entry in attachment:
+ if entry['pathname'].endswith(".log"):
+ entry['mime_type'] = "text/plain"
results = self._get_test_results()
return {
'test_id': 'lava-command',
'analyzer_assigned_date': now,
- 'analyzer_assigned_uuid': str(uuid4()),
+ 'analyzer_assigned_uuid': self._uuid,
'time_check_performed': False,
'test_results': results,
'attachments': attachment
diff --git a/lava_dispatcher/config.py b/lava_dispatcher/config.py
index 63aa3c6..10bccc8 100644
--- a/lava_dispatcher/config.py
+++ b/lava_dispatcher/config.py
@@ -29,6 +29,11 @@ from configglue import parser, schema
class DeviceSchema(schema.Schema):
+
+ # Host-side hook for lava_command_run
+ host_hook_enter_command = schema.StringOption()
+ host_hook_exit_command = schema.StringOption()
+
master_testboot_dir = schema.StringOption()
master_testboot_label = schema.StringOption()
master_testrootfs_dir = schema.StringOption()
--
2.5.0
> From: wangbo <wangbo.hfut(a)qq.com>
> Date: Thu, 28 Jan 2016 20:39:25 +0800
> Subject: LAVA uninstall and upgrade, build method from the source code
> Hi all,
>
> I installed LAVA on Ubuntu 14.04 in our lab, but later we plan to upgrade the system to Ubuntu 15.10, so what should I do now?
I'm afraid the only long term option is to migrate to Debian Jessie,
possibly using a VM, and then using jessie-backports. It is uncertain
whether it will be possible to support LAVA on the upcoming Ubuntu
Xenial 16.04LTS and support for Trusty has already been frozen. There
have been problems upgrading from 14.04 - these issues have already
been discussed on the lists.
https://lists.linaro.org/pipermail/lava-announce/2015-November/000003.html
The next upload of LAVA should be able to migrate into Ubuntu ahead of
the cut off for the Xenial release but that is outside our control. It
is also unknown whether it will be possible to maintain a system based
on Xenial - the LAVA software team will be unable to maintain such
support and users would need to be proactive in fixing problems that
may arise.
> Because I have already installed LAVA, so should I uninstall it firstly, and then use apt-get install to reinstall LAVA? But I can't find more information about this in the website, can you give me some advice, thanks very much.
https://staging.validation.linaro.org/static/docs/installing_on_debian.html…
The main problems will be with the database, so building in any way
will have the same difficulties. Trusty is, sadly, a dead end at the
moment.
> And, how to install LAVA from the source code?
Not supported - installation is only supported as Debian packages.
LAVA is much more than a pip install / virtualenv can ever manage.
> LAVA includes different parts, like lava-server, lava-dispatcher, lava-tool, etc. So which one should be installed firstly? By the way in the Git, for example "https://git.linaro.org/lava/lava-server.git/tree", there is no document about the install method. Now, I will modify some code to meet our demands in our lab, and then replace the corresponding file. We plan to manage the code with Git too. So if we know the install method from the source code, it will be better to build the code.
There is no documentation for installing from git because the
installation method has moved to using packages. We abandoned the old
install from git methods a long time ago due to interminable bugs and
insolvable dilemmas.
During the migration to the new dispatcher, things are too complex to
support more than Debian Jessie (and the next Debian release,
Stretch). Once the migration is complete (in 2017), the advantages of
the new design should also make it easier to support other operating
systems. We are a long way from that at the moment.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Hi Neil et al.
I'd like to have the host record power measurements during a job, or
during a unit-test.
browsing the lava manual, I came across the shell-hooks which seem to
enable what I want to do.
Suppose I want to call a script "capture-start.sh" at the top of the
job, and "capture-stop.sh" at the end, is this the proper way to do stuff:
In the job json:
-------------------
```
{
"command": "lava_test_shell",
"parameters": {
"testdef_repos": [
{
"git-repo": "https://github.com....
/lava-test-definitions.git",
"testdef": "capture-start.yaml"
}
],
"timeout": 1800
}
},
.... dispatcher actions of interest ....
{
"command": "lava_test_shell",
"parameters": {
"testdef_repos": [
{
"git-repo": "https://github.com....
/lava-test-definitions.git",
"testdef": "capture-stop.yaml"
}
],
"timeout": 1800
}
},
```
In the yaml, do I need to override all the handlers ? If I don't need
end_case or postprocess_test_result do I need to fill them, and point to
a "nop.sh" script "?
I assume that I need to provide at lest one step to be done on the
target, even if I am only interested in the host-side commands, would
the following yaml make sense:
```
metadata:
format: Lava-Test Test Definition 1.0
name: power-capture-start
run:
steps:
- lava-test-case pass-test --shell echo starting power measurements
handler:
handler-name: shell-hooks
params:
device_config_vars:
DEVICE_TYPE: device_type
handlers:
start_testcase: capture-start-start-hook.sh
end_testcase: nop.sh
postprocess_test_result: nop.sh
```
Thanks and Regards,
Marc.
Hi all,
I have a basic practical question: I'm trying to reset a panda-es board
with a suitable master image for lava to use with the default
device-types confs, namely with:
lava-dispatcher/device-types/panda-es.conf:bootloader_prompt = U-Boot
so I wrote panda-master-20150303.img.tgz (linaro master-images) to the
sdcard I'm using.
I'd expect u-boot.bin on the boot partition to being compatible with the
expected lava prompt and u-boot command set ('bootz' supported for
instance), but I keep having an incompatible config of u-boot ("Panda"
as a prompt, bootz not supported, etc...), what am I missing?
Thanks and Regards,
Marc.
Hi Neil et al,
I'm trying to debug a simple qemu job that goes straight from running to
incomplete without log creation (used to working ok, but I reinstalled
everything on a different machine...)
Looking at /var/log/lava-server/lava-scheduler.log I see the following:
2015-12-09 15:22:27,838 [INFO] [lava_scheduler_daemon.job.JobRunner.14]
starting job {u'timeout': 18000, 'health_check': False, u'job_name':
u'qemu-arm-test', u'actions': [{u'command': u'deploy_linaro_kernel',
u'parameters': {u'login_prompt': u'login:', u'kernel': u'
http://images.validation.linaro.org/functional-test-images/qemu-arm/zImage-…',
u'username': u'root', u'rootfs': u'
http://images.validation.linaro.org/functional-test-images/qemu-arm/core-im…'}},
{u'command': u'boot_linaro_image', u'parameters': {u'test_image_prompt':
u'root@qemu-system-arm:~#'}}], u'target': u'qemu0'}
2015-12-09 15:22:27,838 [INFO] [lava_scheduler_daemon.job.MonitorJob]
monitoring "setsid lava-server manage schedulermonitor 14 lava-dispatch
qemu0 /tmp/tmpPd4nGs -l info -f /var/log/lava-server/lava-scheduler.log"
2015-12-09 15:22:29,171 [INFO] [lava_scheduler_daemon.job.Job.qemu0]
executing "lava-dispatch /tmp/tmpFltuQQ --output-dir
/var/lib/lava-server/default/media/job-output/job-14"
2015-12-09 15:22:30,388 [INFO]
[lava_scheduler_daemon.job.DispatcherProcessProtocol] childConnectionLost
for qemu0: 0
2015-12-09 15:22:30,389 [INFO]
[lava_scheduler_daemon.job.DispatcherProcessProtocol] childConnectionLost
for qemu0: 1
2015-12-09 15:22:30,389 [INFO]
[lava_scheduler_daemon.job.DispatcherProcessProtocol] childConnectionLost
for qemu0: 2
2015-12-09 15:22:30,389 [INFO]
[lava_scheduler_daemon.job.DispatcherProcessProtocol] processExited for
qemu0: A process has ended with a probable error condition: process ended
with exit code 1.
2015-12-09 15:22:30,389 [INFO]
[lava_scheduler_daemon.job.DispatcherProcessProtocol] processEnded for
qemu0: A process has ended with a probable error condition: process ended
with exit code 1.
2015-12-09 15:22:30,389 [INFO] [lava_scheduler_daemon.job.Job.qemu0] job
finished on qemu0
2015-12-09 15:22:30,389 [INFO] [lava_scheduler_daemon.job.Job.qemu0] job
incomplete: reported 1 exit code
2015-12-09 15:22:30,422 [INFO]
[lava_scheduler_daemon.dbjobsource.DatabaseJobSource] job 14 completed on
qemu0
I tried to run manually:
setsid lava-server manage schedulermonitor 14 lava-dispatch qemu0
qemu-arm.json
powerci@lab-baylibre:~/POWERCI/scripts/user$ 2015-12-09 15:23:23,285
[ERROR] [lava_scheduler_daemon.job.Job.qemu0] AttributeError: 'Job' object
has no attribute '_protocol'
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line
1203, in mainLoop
self.runUntilCurrent()
File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line
798, in runUntilCurrent
f(*a, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line
393, in callback
self._startRunCallbacks(result)
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line
501, in _startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line
588, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/usr/lib/python2.7/dist-packages/lava_scheduler_daemon/job.py",
line 226, in _run
self.cancel(exc)
File "/usr/lib/python2.7/dist-packages/lava_scheduler_daemon/job.py",
line 157, in cancel
self._protocol.transport.signalProcess(getattr(signal, signame))
exceptions.AttributeError: 'Job' object has no attribute '_protocol'
Note that I get the same issue with other jobs (boards, kvm): submission is
OK, but incomplete, and no log.
Any help would be much appreciated!
Many thanks,
Marc.
Hi,
is it possible to interact with a host tool (running on the dispatcher
host) to gather arbitrary key values and post them into the test result
stream. IOW, How do I add a MEASUREMENT/UNITS values using the output of
a host tool (no target shell interaction) ?
Thanks,
Marc.
Hi Neil et al.
I'm working at add my switching device to lavapdu deamon, so I pull Matt's
github etc...but when trying to restart the daemon I get the following
error:
déc. 04 15:56:49 powerci-lava-disp lavapdu-runner[3081]: Reading
settings from /etc/lavapdu/lavapdu.conf
déc. 04 15:56:49 powerci-lava-disp lavapdu-runner[3081]: Traceback
(most recent call last):
déc. 04 15:56:49 powerci-lava-disp lavapdu-runner[3081]: File
"/usr/sbin/lavapdu-runner", line 103, in <module>
déc. 04 15:56:49 powerci-lava-disp lavapdu-runner[3081]: context =
daemon.DaemonContext(
déc. 04 15:56:49 powerci-lava-disp lavapdu-runner[3081]:
AttributeError: 'module' object has no attribute 'DaemonContext'
déc. 04 15:56:49 powerci-lava-disp lavapdu-runner[3081]: ...fail!
déc. 04 15:56:49 powerci-lava-disp systemd[1]: lavapdu-runner.service:
Control process exited, code=exited status=1
déc. 04 15:56:49 powerci-lava-disp systemd[1]: Failed to start LSB:
LAVA PDU Runner.
déc. 04 15:56:49 powerci-lava-disp systemd[1]: lavapdu-runner.service:
Unit entered failed state.
déc. 04 15:56:49 powerci-lava-disp systemd[1]: lavapdu-runner.service:
Failed with result 'exit-code'.
I tried to revert to the master branch, re-installed with setup.py
with no success. The json config file seems ok.
Any clue?
Many Thanks,
Marc.
Hi All,
I'm setting up a local LAVA scheduler+dispatcher (standalone).
I'm able to posts jobs, but I think I'm missing an initial configuration
for some actions to complete, namely
action deploy_linaro_image
the eMMC flash content of my BBB-RevB6 is pretty standard (debian-7.9 of
2015-11-03, 2gb image).
Based on the log, the dispatcher expects the default "testboot" and
"testrootfs" partitions with offset 2, I assuime this means that those
partitions must be manually created the first time with index 3 and 4 in
order for this to work.
I would appreciate some hints, to avoid unnecessary test-and-try, what
would be the expected partitionning for BBB with LAVA ?
Thanks in advance,
best regards,
Marc.
Please let us know if you are using OpenID authentication with LAVA.
Newer versions of django will make it impossible to support
django-openid-auth in Debian unstable and testing. The version of
django-openid-auth in Jessie can continue to be used, so we would like
to know how many users want to continue with this support.
OpenID as a protocol has been dying for some time and Linaro has moved
over to LDAP, which is fine if LDAP is already available.
The time pressure for this change is coming from the schedule to get
the latest django and the latest lava packages into Ubuntu Xenial
16.04LTS which means that support needs to be implemented in the
2015.12 or 2016.1 LAVA releases. This is why this is quickly following
the trusty change. We have been aware of the issues with
django-openid-auth for some time, it was only when we had completed
the move of the Cambridge lab to LDAP that changes involving
django-openid-auth could be considered.
If you are using OpenID authentication (e.g. using Launchpad or Google
OpenID), please let us know.
If you would like to see some other forms of authentication supported,
also let us know. We can investigate Python Social Auth
(http://psa.matiasaguirre.net/), if there is interest.
If we don't hear from users who want django-openid-auth support for
use on Debian Jessie, we will drop django-openid-auth support from all
lava builds. This will leave LDAP and local Django accounts in
2015.12.
If anyone has experience of other django authentication modules, also
let us know.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
See also:
https://lists.linaro.org/pipermail/lava-announce/2015-November/000002.html
which was also sent to these lists (except linaro-dev).
So far, nobody has come forward as a Trusty user. The only Trusty
instance of which we are aware is already due to migrate to Debian
Jessie.
The LAVA software team are now applying updates which will freeze LAVA
software support for Ubuntu Trusty at 2015.9 for lava-dispatcher and
2015.9.post1 for lava-server due to the complexities of supporting
both django1.6 and the current django1.7 in Jessie and django1.8,
possibly django1.9 by the time Debian Stretch is released.
The last packages for Ubuntu Trusty 14.04LTS will be:
lava-server 2015.9.post1
lava-dispatcher 2015.9
Once these changes are applied, the Debian packaging used to build
future versions of LAVA packages will prevent builds against django1.6
and prevent installation if django1.6 is found, in order to prevent
database corruption.
This means that Trusty users will not be able to use the results of
the dispatcher refactoring.
Ubuntu Xenial Xerus - which is planned to be the 16.04LTS in April
2016 - is expected to pick up LAVA software releases from Debian up
until the 2016.1 release (possibly 2016.2) and is also expected to be
using django1.8. The next Debian stable release (Stretch), for which
no date has yet been set, may use django1.9.
Initial attempts at migrating a test instance from Trusty to django1.7
did not go well and the migration from Trusty to Xenial cannot be
supported by the LAVA software team - the recommendation is to go
directly from 2015.9 on Trusty to the same version available for
Debian Jessie but there will still be work to be done to prepare and
implement the migration which will be instance-dependent.
Documentation is being added to assist with this migration but there
will remain risks of data loss which will need to be managed for each
instance. It is imperative that anyone using Trusty has an up to date
backup of the postgresql database dump before considering any
migration. If the existing data is to be dropped, a new install on
Debian Jessie is recommended.
It is not possible for the LAVA software team to support all versions
of django from 1.6 to 1.9 - particular problems are known when going
from django1.6 to django1.7 as the methods to migrate the lava-server
database changed fundamentally in django1.7.
Notes are being added to the documentation on the trusty branch based
on 2015.9 to be released within lava-server 2015.9.post1 and to the
documentation in the master branch (which will go into 2015.12).
All future builds of LAVA software will now be made and uploaded only
to Debian and releases.linaro.org.
So far, nobody has come forward who is willing to maintain packaging
for LAVA software on any distribution other than Debian. As the
refactoring proceeds, we expect that it will become easier to package
LAVA for other distributions but the migration to the refactoring must
be complete first.
Everyone interested in or using LAVA is encouraged to subscribe to the
lava-announce mailing list which is low volume and only used for
substantial changes like this.
https://lists.linaro.org/mailman/listinfo/lava-announce
See also https://validation.linaro.org/static/docs/support.html
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
