- lava-users - op-lists.linaro.org

Procedure to install lava-coordinator setup

by Chetan Sharma

Hi Everyone, Can anyone share a step by step process to install Lava-coordinator in existing Setup to execute multinode job. Existing Setup :- 1. Lava-Server (Machine 1) 2. Lava-worker (Machine 2) 3. Lava-worker (Machine 3) -- Thanks & Regards Chetan Sharma

7 years, 11 months

3
4
0 0

lava-tool wait-job-events

by Loys Ollivier

Hello everyone, I am using lava-tool to monitor my jobs. Previously I used: $ lava-tool submit-job --block Using version of lava-tool 0.23 I now have this message: --> This kind of polling is deprecated and will be removed in the next release. Please use "wait-for-job" command. But "wait-for-job" doesn't exist. There is a "wait-job-events" option though. I tried this one and it doesn't return even once the job has finished. If I manually stop it and restart it with the same job number I get as output: --> Job already finished with status Complete. Command I'm using: $ lava-tool wait-job-events --job-id 20 http://user@lava-server Is there anything I'm doing incorrectly ? Or are you aware of this bug ? Thanks ! -- Loys OLLIVIER

7 years, 11 months

3
13
0 0

Booting with Coreboot/Depthcharge

by Guillaume Tucker

A change was sent a while ago to add support for the Coreboot / Depthcharge bootloader which is used on Chromebook devices. This is useful in particular to avoid having to install U-Boot on Chromebook devices. See this Gerrit review here for previous history: https://review.linaro.org/#/c/15203/ I'm now opening this case again to try and get this resolved, there seem to be several issues with the original patch that would need to be clarified. Also, some things might have changed since then in LAVA or Coreboot which could potentially lead to a different approach - any feedback on this would be welcome. To start with, I understand that running mkimage on the dispatcher is not a valid thing to do, it should receive a FIT (flattened image tree) kernel image ready to be booted. This complicates things a bit for projects like kernelci.org where only a plain kernel image is built and ramdisks are served separately, but it's fair enough to say that LAVA is not meant to be packaging kernel images on the fly. Then I believe creating the command line file in LAVA should be fine, although it probably makes more sense to have both the FIT image and cmdline file generated by the same build system. In any case, both files would need to be served from the dispatcher TFTP server to the target device running Coreboot / Depthcharge. So the idea was basically to have an option in Coreboot / Depthcharge to interactively tell it where to find these files for the current job to run, say: <JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/vmlinuz <JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/cmdline It looks like the current patch in Gerrit relies on this location to be hard-coded in the bootloader, which works fine for a private development set-up but not for LAVA. To recap, my understanding is that the "depthcharge" boot support code in LAVA would need to: * maybe create the cmdline file with basically the kernel command line split up with one argument per line * or just download the cmdline file along with the vmlinuz FIT * place both the cmdline and vmlinuz FIT files in the job's TFTP directory on the dispatcher * turn on the device and open the serial console... * interactively pass at least the path to the job TFTP directory on the serial console (and if possible the server IP address as well, and maybe even the individual file names rather than hard-coded vmlinuz and cmdline) * look for a bootloader message to know when the kernel starts to load and hand over to the next action (login...) Please let me know if this sounds reasonable or if we should be doing anything differently. I think it would be good to have some agreement and a clear understanding of how this is going to be implemented before starting to work on the code again. Best wishes, Guillaume

7 years, 11 months

2
17
0 0

LAB Notice: LAVA Instance downtime

by Dave Pigott

[bcc’d to everyone] Hi all, Next week, starting on Monday, we will be upgrading all the LAVA servers to Debian Stretch and then upgrading to LAVA 2018.01. This will involve minimal downtime for each instance. Devices will be off-lined ahead of each upgrade. The order of upgrade will be: Monday: LNG: lng.validation.linaro.org PMWG: pmwg.validation.linaro.org Tuesday: Production: validation.linaro.org LKFT: lkft.validation.linaro.org All being well, the downtime, i.e. the time you will not be able to submit jobs or see the web site, will be of the order of one hour per instance. If that is going to change I will send out a new notification. Thanks for your patience, Dave ---------------- Dave Pigott LAVA Lab Lead Linaro Ltd t: (+44) (0) 1223 400063

7 years, 11 months

1
2
0 0

Re: [Lava-users] No result coming when job submitted in lava

by Robert Marshall

Smita, Yes I looked and confirm that analysis, you've not commented on what the U-Boot prompt is? I'm putting lava-users back on the cc list in case others wish to comment, and making my comment below clearer Robert "Gumansingh, Smita" <Smita_Gumansingh(a)mentor.com> writes: > Hi Robert, > > Have you got any chance to see my bbb health check log > > bbb health check fails at this point > > ------------ ---------------------- > send-reboot-commands timed out after 179 seconds > end: 2.4.1.1 send-reboot-commands (duration 00:02:59) > ---------------------------- > > Please have a look on this log https://pastebin.com/gPmMG84J > > Thanks & Regards, > Smita Gumansingh > > ________________________________________ > From: Robert Marshall <robert.marshall(a)codethink.co.uk> > Sent: Wednesday, January 10, 2018 6:21 PM > To: Gumansingh, Smita > Subject: Re: [Lava-users] No result coming when job submitted in lava > > Smita > > Though you only to replace that line *if* the U-Boot prompt doesn't ^ need > consist of "U-Boot⇒" - interrupt the BBB boot to see what it actually is, > and if it is necessary, you also need to remove the other line - > apologies for the instructions not being clear here. > > Thanks for the fuller output in the other email! > > Robert > > "Gumansingh, Smita" <Smita_Gumansingh(a)mentor.com> writes: > >> Thanks Robert for the quick response >> >> Currently u-boot is showing in /etc/lava-server/dispatcher-config/device-types/beaglebone-black.jinja2 is: >> >> {% set bootloader_prompt = bootloader_prompt|default('U-Boot') %} >> >> As you suggested I added the line: {% set bootloader_prompt = bootloader_prompt|default('⇒') %} and submitted a job >> >> No output from the job when submitted... >> >> Thanks & Regards, >> Smita Gumansingh >> >> ________________________________________ >> From: Robert Marshall <robert.marshall(a)codethink.co.uk> >> Sent: Wednesday, January 10, 2018 5:08 PM >> To: Gumansingh, Smita >> Cc: lava-users(a)lists.linaro.org >> Subject: Re: [Lava-users] No result coming when job submitted in lava >> >> Hi, some comments below! >> >> "Gumansingh, Smita" <Smita_Gumansingh(a)mentor.com> writes: >> >>> Hi, >>> I am new to lava and trying to submit a job on lava scheduler ,job submitted nut no result is coming. I am trying to >>> test the CIP Kernel on the Beaglebone Black(board is physically connected to my linux machine). Health checkup is >>> working somehow . >> >> By 'somehow' do you mean the health check is completing correctly and the >> device is shown as online? >> >>> I am following the steps from here >>> https://wiki.linuxfoundation.org/civilinfrastructureplatform/ciptestingrefe… >>> >>> Pre-requise: >>> 1. I have built(cip_v4.4.92) the CIP Kernel with Kernel CI as the steps mentioned in >>> https://wiki.linuxfoundation.org/civilinfrastructureplatform/cipsystembuild… >>> 2.Target is booted and up and flashed with debian 4.9 rootfs >>> >> >> What is the U-Boot prompt with this version, if it is ⇒ rather than U-Boot⇒ >> >> on the vagrant machine you need to >> >> sudo vi /etc/lava-server/dispatcher-config/device-types/beaglebone-black.jinja2 >> And add the line: {% set bootloader_prompt = bootloader_prompt|default('⇒') %} >> >> >>> Job Definination is pasted here >>> >>> https://pastebin.com/YwnPXidK >>> >>> Need help for going further !!!! >> >> Do you get any output from the job? >> >>> >>> Thanks & Regards, >>> Smita Gumansingh >>> >> >> Robert

7 years, 11 months

1
0
0 0

No result coming when job submitted in lava

by Gumansingh, Smita

Hi, I am new to lava and trying to submit a job on lava scheduler ,job submitted nut no result is coming. I am trying to test the CIP Kernel on the Beaglebone Black(board is physically connected to my linux machine). Health checkup is working somehow . I am following the steps from here https://wiki.linuxfoundation.org/civilinfrastructureplatform/ciptestingrefe… Pre-requise: 1. I have built(cip_v4.4.92) the CIP Kernel with Kernel CI as the steps mentioned in https://wiki.linuxfoundation.org/civilinfrastructureplatform/cipsystembuild… 2.Target is booted and up and flashed with debian 4.9 rootfs Job Definination is pasted here https://pastebin.com/YwnPXidK Need help for going further !!!! Thanks & Regards, Smita Gumansingh

7 years, 11 months

2
1
0 0

Better bootloader failure detection

by Matt Hart

Currently it is difficult to tell the difference between an infrastructure problem in a device bootloader, or a kernel failure. If a kernel silently fails to boot, LAVA throws a bootloader-commands timeout because it hasn’t matched ‘Linux version’ to know the kernel has started. However, this timeout could also be caused by a real problem in the bootloader, such as a DHCP failure or a TFTP timeout. KernelCI would like to catch actual infrastructure problems in the bootloader, but can’t tell if the kernel just didn’t boot, or the commands actually timed out in the bootloader. To fix this, we're going to: - change the bootloader-commands action to finish when it has sent the last command - have auto-login-action takeover monitoring the kernel boot process - extend bootloader-commands to match more infrastructure problems - update uboot commands to execute the commands in order (like the other bootloader implementations), rather than building a script and then calling that as the last command This work is scoped for the January 2018.1 LAVA release.

7 years, 11 months

3
2
0 0

Job stuck

by Thomas Petazzoni

Hello, We have an installation where we use LAVA 2017.12. We are regularly seeing jobs that remain stuck for several days. For example, I have a job right now on the Armada XP GP that is stuck since 1 day and 11 hours. The log visible in the LAVA Web interface looks like this: http://code.bulix.org/7pvru8-255308?raw This is job #855671 in our setup. The logs on the lava-slave look like this: http://code.bulix.org/c5tejy-255312?raw So, from the lava-slave point of view, the job is finished. However, the "Job END" message had to be resent several times to the master. Interestingly, this sequence lead to a very nice: ERROR [855671] lava-run crashed On the lava-master side (which runs on another machine), the logs look like this: http://code.bulix.org/b61keb-255316?raw And this happens for lots of jobs. Pretty much every day or two, we have ten boards stuck in this situation. I have the lava-master logs with DEBUGs if this can be helpful. However, must DEBUG logs don't have the job number in them, so it makes it difficult to associate the DEBUG messages with the problematic job (since numerous other jobs are running). Does anyone has an idea what could be causing this ? Or how to debug this further ? Best regards, Thomas Petazzoni -- Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com

7 years, 11 months

3
2
0 0

LXC device not working in multinode jobs

by Andrei Gheorghiu

Hi, I am having issues using an LXC device within the multinode LAVA protocole/API. The job yaml gets validated, but it fails to run with the following error: Missing protocol 'lava-lxc' in ['lava-multinode'] Full yaml used can be found here: https://pastebin.com/BUsX0G0C While searching for a fix I also found this email from a year ago, which wasn't answered (seems related): http://linaro-validation.linaro.narkive.com/mXhxhHqy/issues-with-lava-multi… Thanks, Andrei

7 years, 11 months

2
1
0 0

Help: using custom script - run.sh error

by Mylene JOSSERAND

Hello everyone, I am trying to use lava with a custom script[1]. Previously, my test was used as a shell script (using exit) and I updated it to be a custom script to be able to use "skip" result. I ran a job and I got this error: /lava-854766/0/tests/1_custom-tests/run.sh: line 1: ####: not found It seems to be linked with my echo of "#### Starting NAND test ####" [2]. Here is the yaml used: https://github.com/free-electrons/test_suite/blob/master/tests/custom.yaml the source of the script: https://github.com/free-electrons/test_suite/blob/master/scripts/nand.sh the jinja file: https://github.com/free-electrons/custom_tests_tool/blob/master/src/jobs_te… and some part of the output when the test ran and failed: http://code.bulix.org/ca3mc7-254957?raw As I am new to LAVA, I have probably missed something. On the current test example [3], it seems that there is no output using "echo". Is it possible that my "echo" leads to this error? It is possible to print messages in a custom script, right? [1]: https://validation.linaro.org/static/docs/v2/writing-tests.html#writing-cus… [2]: https://github.com/free-electrons/test_suite/blob/master/scripts/nand.sh#L9 [3]: https://git.linaro.org/lava-team/refactoring.git/tree/functional/unittests.… Thank you in advance for your help, Best regards, -- Mylène Josserand, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com

7 years, 11 months

2
4
0 0

Need Help With Simplistic Testing Over SSH

by Tan, Jian Chern

Hi, I'm trying to get started with LAVA by first attempting to do some simplistic testing over SSH to run some tests on a device that doesn't have a default template. I'm getting a few connection errors and others like output ['Permission denied (publickey,password).\r', 'lost connection', ''] and it's probably because I've misconfigured the jinja files, having little experience with these LAVA jinja templates. Attached are the job logs, jinja2 template files and the test yaml file. Could anyone point me in the right direction by either providing sample ssh jinja files and job files or pointing out the errors in my config. Thanks! Thanks! Jian Chern

7 years, 11 months

1
0
0 0

Unpacking tar.bz2 root filesystems seems to malfunction

by Dragos Iorgulescu

Hi all! I am working with some Raspberry Pi boards and, while defining some particularities for booting via NFS, I came across an issue regarding the way the rootfs file is unpacked. To be more specific, after looking through the LAVA code, it seems that the untar_file() method runs on the conditional branch that states the tar members are specified. I do not understand how this implementation is designed in regards to tar archive manipulation, but the following scenario takes place: * The rootfs archive we specify in the job definition is copied and renamed, the extension being changed from .tar.bz2 to a plain .tar (which seems a bit strange tom e, on its own) * Then, after unpacking to the LAVA temp job dir, only two directories are extracted from the whole rootfs archive. I did manage to „inverveene” while the job was running and looking in the temp dir to see exactly what gets extracted. Since the rest of the folders from the rootfs are not available, the job fails once the system starts booting While investigating, I noticed that the untarf_file() method is invoked from download.py (/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/actions/deploy/) - @ line 344 – with members being specified. I know that, because when I manually changed how this call is made and clearly specified “None” for the “member” positional argument, I get the following error at job runtime: https://paste.debian.net/1001252/ (the full job log can be found here: https://paste.debian.net/1001253/ ) The code changes I am talking about (and the code where I suspect something is ambiguous) are pointed out here: https://paste.debian.net/1001256/ . A job definition we use for this board integration can be analyzed here: https://paste.debian.net/1001257/ The full log of the initial job, that ran without my code changes, can be found here: https://paste.debian.net/1001254/ . The „kernel panic” message ocurs because, as stated in one of the errors, the “init” folder is missing, which is accurate, because only bin & dev are extracted from our rootfs tar.bz2 archive. What could we do? Is this something that needs to be adjusted/fixed in LAVA? Kind regards, Dragoș

7 years, 12 months

4
13
0 0

[RFC][PATCH] lava_dispatcher/actions/boot: Add /dev/loopN device to LXC

by Aníbal Limón

From: Aníbal Limón <anibal.limon(a)linaro.org> Now the Test writer has access to the images inside the LXC to make changes previous deploy/flash into the board, in order to support mount/modify rootfs images the loop device is needed. Add a parameter in the lxc-boot action to map a free loop device (losetup -f) into the LXC. Change-Id: I7060ebac12b10e5390560da082fe6c49568c5ffc Signed-off-by: Aníbal Limón <anibal.limon(a)linaro.org> --- lava_dispatcher/actions/boot/lxc.py | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/lava_dispatcher/actions/boot/lxc.py b/lava_dispatcher/actions/boot/lxc.py index d896d303..e3e3cb48 100644 --- a/lava_dispatcher/actions/boot/lxc.py +++ b/lava_dispatcher/actions/boot/lxc.py @@ -75,7 +75,11 @@ class BootLxcAction(BootAction): def populate(self, parameters): self.internal_pipeline = Pipeline(parent=self, job=self.job, parameters=parameters) self.internal_pipeline.add_action(LxcStartAction()) - self.internal_pipeline.add_action(LxcAddStaticDevices()) + + lxc_add_loop = False + if 'lxc_add_loop' in parameters: + lxc_add_loop = parameters.get('lxc_add_loop', False) + self.internal_pipeline.add_action(LxcAddStaticDevices(lxc_add_loop)) self.internal_pipeline.add_action(ConnectLxc()) # Skip AutoLoginAction unconditionally as this action tries to parse kernel message # self.internal_pipeline.add_action(AutoLoginAction()) @@ -91,11 +95,12 @@ class LxcAddStaticDevices(Action): worker. """ - def __init__(self): + def __init__(self, lxc_add_loop=False): super(LxcAddStaticDevices, self).__init__() self.name = 'lxc-add-static' self.description = 'Add devices which are permanently powered by the worker to the LXC' self.summary = 'Add static devices to the LXC' + self.lxc_add_loop = lxc_add_loop def validate(self): super(LxcAddStaticDevices, self).validate() @@ -115,6 +120,15 @@ class LxcAddStaticDevices(Action): def run(self, connection, max_end_time, args=None): connection = super(LxcAddStaticDevices, self).run(connection, max_end_time, args) lxc_name = self.get_namespace_data(action='lxc-create-action', label='lxc', key='name') + + if self.lxc_add_loop: + lxc_get_loop_cmd = ['losetup', '-f'] + loop_device = self.run_command(lxc_get_loop_cmd, allow_silent=True).strip() + lxc_loop_cmd = ['lxc-device', '-n', lxc_name, 'add', loop_device] + cmd_out = self.run_command(lxc_loop_cmd) + if cmd_out: + self.logger.debug(cmd_out) + # If there is no static_info then this action should be idempotent. if 'static_info' not in self.job.device: return connection -- 2.11.0

7 years, 12 months

2
2
0 0

Re: [Lava-users] Booting with Coreboot/Depthcharge

by Tomeu Vizoso

On 18 Dec 2017 3:45 p.m., "Guillaume Tucker" <guillaume.tucker(a)collabora.com> wrote: On 18/12/17 11:45, Neil Williams wrote: > On 14 December 2017 at 09:47, Guillaume Tucker < > guillaume.tucker(a)collabora.com> wrote: > > On 07/12/17 17:16, Neil Williams wrote: >> >> On 7 December 2017 at 16:20, Guillaume Tucker < >>> guillaume.tucker(a)collabora.com> wrote: >>> >>> A change was sent a while ago to add support for the Coreboot / >>> >>>> Depthcharge bootloader which is used on Chromebook devices. This >>>> is useful in particular to avoid having to install U-Boot on >>>> Chromebook devices. See this Gerrit review here for previous >>>> history: >>>> >>>> https://review.linaro.org/#/c/15203/ >>>> >>>> I'm now opening this case again to try and get this resolved, >>>> there seem to be several issues with the original patch that >>>> would need to be clarified. Also, some things might have changed >>>> since then in LAVA or Coreboot which could potentially lead to a >>>> different approach - any feedback on this would be welcome. >>>> >>>> >>>> Thanks for picking this up. >>> >>> >> You're welcome. I've now uploaded a new version which generates >> the command line file but not the FIT image, it expects the >> kernel image to be already in this format. Still the same >> Gerrit number: >> >> https://review.linaro.org/#/c/15203/ >> >> I've also made a patch to add the rk3288-veyron-jaq as >> a "depthcharge" device type: >> >> https://review.linaro.org/#/c/22992/ >> >> So as a next step, it would be convenient to find a way to have >> the FIT image generated as part of the LAVA job with a given >> kernel image, dtb, maybe the .its file and optionally a ramdisk. >> >> For the reference: >> >> http://git.denx.de/?p=u-boot.git;a=blob;f=doc/uImage.FIT/how >> to.txt;hb=master >> >> To start with, I understand that running mkimage on the >> >>> dispatcher is not a valid thing to do, it should receive a >>>> FIT (flattened image tree) kernel image ready to be booted. This >>>> complicates things a bit for projects like kernelci.org where >>>> only a plain kernel image is built and ramdisks are served >>>> separately, but it's fair enough to say that LAVA is not meant to >>>> be packaging kernel images on the fly. >>>> >>>> >>>> We've come up with a method in the meantime, although it does mean using >>> LXC but that makes it completely generic. It's principally designed for >>> boards which need to munge a kernel and other files into an image to be >>> transferred to the device using tools like fastboot. This is how KernelCI >>> will be able to submit boot tests on devices like HiKey and db410c. >>> Sadly, >>> the example test job is suffering because the db410c devices have a >>> different problem which is keeping them offline. Matt has been looking >>> into >>> this. >>> >>> https://staging.validation.linaro.org/scheduler/job/203317/definition >>> >>> https://staging.validation.linaro.org/static/docs/v2/actions >>> -deploy.html#index-25 >>> >>> >> Thanks for the pointers, seems worth investigating. >> >> On the other hand, creating the FIT image is a similar process to >> that of uImage, which is currently being done directly on the >> dispatcher: >> >> https://git.linaro.org/lava/lava-dispatcher.git/tree/lava_di >> spatcher/actions/deploy/prepare.py#n79 >> >> So would it make sense to add some code there to support FIT? >> > > > What is an example command line to mkimage to do this? > mkimage -D "-I dts -O dtb -p 2048" -f rk3288-veyron-jaq.its arch/arm/boot/vmlinuz Is the its file really needed? I added the ramdisk parameter precisely so lava doesn't need to generate one. Regards, Tomeu Are any external configuration files required? > Everything should be in the .its file, and it should also be possible to generate it on the fly using a template and the LAVA device properties (kernel load address etc...). If this proves to not be flexible enough in practice, then I suppose the .its file could be downloaded although I think we should avoid doing this if we can. Then I believe creating the command line file in LAVA should be >> >>> fine, although it probably makes more sense to have both the FIT >>>> image and cmdline file generated by the same build system. In >>>> any case, both files would need to be served from the dispatcher >>>> TFTP server to the target device running Coreboot / Depthcharge. >>>> >>>> >>>> That bit is fine, the problem is why this cannot use the existing >>> temporary >>> paths which all the other TFTP devices use. Is it just to do some >>> mangling >>> of the files? >>> >>> >> This is resolved now with the version I sent yesterday. >> > > > That makes this review much better, thanks. > Great, thanks for confirming. So the idea was basically to have an option in Coreboot / >> >>> Depthcharge to interactively tell it where to find these files >>>> for the current job to run, say: >>>> >>>> <JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/vmlinuz >>>> <JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/cmdline >>>> >>>> It looks like the current patch in Gerrit relies on this location >>>> to be hard-coded in the bootloader, which works fine for a >>>> private development set-up but not for LAVA. >>>> >>>> >>>> That makes very little sense because the whole point of TFTP is that >>> everything after the SERVER_IP is just a relative path from the TFTP base >>> directory which is handled by the TFTP daemon itself. >>> >>> >> Ditto. >> >> To recap, my understanding is that the "depthcharge" boot support >> >>> code in LAVA would need to: >>>> >>>> * maybe create the cmdline file with basically the kernel >>>> command line split up with one argument per line >>>> >>>> >>>> Alternatively, do whatever operations are required in a test shell in >>> the >>> LXC and then pass those files to the device - entirely within the test >>> shell support. >>> >>> >> That, or maybe run mkimage on the dispatcher like for uImage... >> >> The cmdline file is now generated on the dispatcher. >> >> * or just download the cmdline file along with the vmlinuz FIT >> >>> >>>> >>> The ready-made FIT kernel image is now downloaded with the >> version I sent yesterday. >> >> * place both the cmdline and vmlinuz FIT files in the job's >> >>> TFTP directory on the dispatcher >>>> >>>> * turn on the device and open the serial console... >>>> >>>> * interactively pass at least the path to the job TFTP >>>> directory on the serial console (and if possible the server >>>> IP address as well, and maybe even the individual file names >>>> rather than hard-coded vmlinuz and cmdline) >>>> >>>> >>>> Isn't this equivalent to what U-Boot already does with TFTP? >>> >>> >> Almost. This part is now all implemented in the last patch I >> sent. One thing though is that the NFS rootfs parameters are >> stored in the kernel cmdline file and not set interactively in >> the bootloader shell. >> > > > How can these be extended by test writers? We do see requests to add > arguments to the NFS parameters but adding options to the kernel command > line itself is all but essential for most testing. > This can be done using the {{ extra_kernel_args }} template variable, see the other change to add base-depthcharge.jinja2: https://review.linaro.org/#/c/22992/1/lava_scheduler_app/tes ts/device-types/base-depthcharge.jinja2 If anything more special ever needs to be done with some parameters such as inserting some IP address, it can be done in DepthchargeCommandOverlay where the command line file is generated. The only command sent is to start the tftp >> boot with the server IP and the relative paths to the kernel and >> cmdline files. >> > On this topic, the changes to add the tftpboot command in Depthcharge are still under review: https://chromium-review.googlesource.com/c/chromiumos/platfo rm/depthcharge/+/451382 So I think it would actually be wiser to not merge base-depthcharge.jinja2 until the review above has been merged in case the command line syntax needs to be adjusted. * look for a bootloader message to know when the kernel starts >> >>> to load and hand over to the next action (login...) >>>> >>>> >>> Done as well, I've now got the veyron-jaq device booting fine >> with NFS rootfs. There was an issue with adding a ramdisk to the >> FIT image as it was to big to boot on the device, will >> investigate this part to add "ramdisk" boot commands. >> >> >> Please let me know if this sounds reasonable or if we should be >> >>> doing anything differently. I think it would be good to have >>>> some agreement and a clear understanding of how this is going to >>>> be implemented before starting to work on the code again. >>>> >>> Best wishes, Guillaume _______________________________________________ Lava-users mailing list Lava-users(a)lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users

7 years, 12 months

2
2
0 0

about the grub parameter : interrupt_prompts

by Xu, Hongyu

Hi , In our testing, we must test the os centos and Ubuntu, but the grub interrupt_prompts is different betweent Ubuntu and centos. So in my device_type template I use "menu_options: {{ grub_method }}", but the "grub_method" must define in device jinjia2 file. I don't want to often chang the device jinja2 file , whether I can define some options in the job file? Please give me some help! The attachmen are my device_type and device file! methods: grub: menu_options: {{ grub_method }} parameters: {% if grub_method == 'centos' %} interrupt_prompt: {{ grub_interrupt_prompt|default('Press \'e\' to edit the selected item, or \'c\' for a command prompt.') }} {% elif grub_method == 'ubuntu' %} interrupt_prompt: {{ grub_interrupt_prompt|default(' Press enter to boot the selected OS') }} {% elif grub_method == 'pxe' %} interrupt_prompt: {{ grub_interrupt_prompt|default('Press \'e\' to edit the selected item, or \'c\' for a command prompt.') }} {% endif %} bootloader_prompt: {{ grub_efi_bootloader_prompt|default('grub>') }} boot_message: {{ kernel_boot_message | default("Booting Linux Kernel...") }} Best Regards XuHongyu This email is intended only for the named addressee. It may contain information that is confidential/private, legally privileged, or copyright-protected, and you should handle it accordingly. If you are not the intended recipient, you do not have legal rights to retain, copy, or distribute this email or its contents, and should promptly delete the email and all electronic copies in your system; do not retain copies in any media. If you have received this email in error, please notify the sender promptly. Thank you.

7 years, 12 months

2
1
0 0

Connection closed by foreign host

by Philippe BEGNIC

Hello Lava Team, We faced to errors when we launched long tests ( e.g LTP tests, stress tests ... ) since we use Lava dockers. The following messages are returned by Lava and the test stop : Connection closed by foreign host. err: lava_test_shell connection dropped Marking unfinished test run as failed These error messages appears during the test, if no message are generated on the console during approximatively six minutes. We found a workaround that consist to send periodically message to the console, allowing the test to complete. I would like to know if we can setup a setting to I would like to know if there is a way to inhibit this control, or to change the timeout settings of this control ? Our configuration : Lava 2017.6 with dockers Regards Philippe Begnic

8 years

3
3
0 0

Troubles using custom notification client

by David Lewin

Hi, >From a client side, I'd like to be notified that jobs are achieved in a distant lava server. I started with the example provided at https://validation.linaro.org/static/docs/v2/data-export.html#write-your-ow… 1. In a 1st try I've changed the example - because I'm using port 10080 - and it works without the <lookup_publisher> . For that, I've hard coded the url returned by lookup_publisher but this only prints out one status at a time, ie :I needed to restart the script each time to have updates : "Submited"->*Ctrl-C*->Running->*Ctrl-C*->complete 2. In a 2nd time I've tried to implement lookup_publisher thinking the status would be updated automatically. In that case it wants to connect to 5500 and obviously fails after timeout import xmlrpclib > > import argparse > import yaml > import signal > import zmq > import xmlrpclib > from urlparse import urlsplit > > > FINISHED_JOB_STATUS = ["Complete", "Incomplete", "Canceled"] > > token = "mytoken" > username = "username" > #hostname = "lava-server:10080" > > > class JobEndTimeoutError(Exception): > """ Raise when the specified job does not finish in certain timeframe. > """ > > > class Timeout(object): > """ Timeout error class with ALARM signal. Accepts time in seconds. """ > class TimeoutError(Exception): > pass > > def __init__(self, sec): > self.sec = sec > > def __enter__(self): > signal.signal(signal.SIGALRM, self.timeout_raise) > signal.alarm(self.sec) > > def __exit__(self, *args): > signal.alarm(0) > > def timeout_raise(self, *args): > raise Timeout.TimeoutError() > > > class JobListener(object): > > def __init__(self, url): > self.context = zmq.Context.instance() > self.sock = self.context.socket(zmq.SUB) > > self.sock.setsockopt(zmq.SUBSCRIBE, b"") > self.sock.connect(url) > > def wait_for_job_end(self, job_id, timeout=None): > > try: > with Timeout(timeout): > while True: > msg = self.sock.recv_multipart() > try: > (topic, uuid, dt, username, data) = msg[:] > except IndexError: > # Droping invalid message > continue > > data = yaml.safe_load(data) > if "job" in data: > if data["job"] == job_id: > if data["status"] in FINISHED_JOB_STATUS: > return data > > except Timeout.TimeoutError: > raise JobEndTimeoutError( > "JobListener timed out after %s seconds." % timeout) > > def lookup_publisher(hostname): > """ > Lookup the publisher details using XML-RPC > on the specified hostname. > """ > xmlrpc_url = "http://%s:10080/RPC2" % (hostname) > server = xmlrpclib.ServerProxy(xmlrpc_url) > socket = server.scheduler.get_publisher_event_socket() > port = urlsplit(socket).port > listener_url = 'tcp://%s:%s' % (hostname,port) > print("Using %s" % listener_url) > return listener_url > > > > if __name__ == '__main__': > # timeout=1200 > > parser = argparse.ArgumentParser() > parser.add_argument("-j", "--job-id", type=int, help="Job ID to wait > for") > parser.add_argument("-t", "--timeout", type=int, help="Timeout in > seconds") > parser.add_argument("--hostname", help="hostname of the instance") > > options = parser.parse_args() > > # server = xmlrpclib.ServerProxy("http://%s:%s@%s/RPC2" % (username, > token, hostname)) > > #print(server.system.listMethods()) > # ret_status=server.scheduler.job_status(options.job_id) > # print (ret_status['job_status']) > > #publisher = 'tcp://%s' % (hostname) > publisher = lookup_publisher(options.hostname) > > listener = JobListener(publisher) > listener.wait_for_job_end(options.job_id, options.timeout) > >

8 years

2
1
0 0

help: can't use specific branch of a test definition repository

by Xu, Hongyu

Hi all , I have saw the doc about : https://validation.linaro.org/static/docs/v2/test-repositories.html#test-re…! My test-definition is in the brach dev-br1, not the master . So I point the branch in my yaml file, but it do noting ! Why? Please give me some help? Best Regards XuHongyu This email is intended only for the named addressee. It may contain information that is confidential/private, legally privileged, or copyright-protected, and you should handle it accordingly. If you are not the intended recipient, you do not have legal rights to retain, copy, or distribute this email or its contents, and should promptly delete the email and all electronic copies in your system; do not retain copies in any media. If you have received this email in error, please notify the sender promptly. Thank you.

8 years

2
3
0 0

Problem with LAVA 2017.12

by LABBE Corentin

Hello I have installed a fresh 2017.12 in a docker and I hit two problem: - With the exception of lava-coordinator, all services (lava-master, lava-server, lava-slave, etc...) fail to start with: service lava-xxxx start lava-xxxx: unrecognized service All Other service successfully start (postgresql, apache2) - I have then re-added old init scripts and when I add a worker with: "lava-server manage workers add lab-slave" I get: Traceback (most recent call last): File "/usr/bin/lava-server", line 78, in <module> main() File "/usr/bin/lava-server", line 74, in main execute_from_command_line(django_options) File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 367, in execute_from_command_line utility.execute() File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 359, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 294, in run_from_argv self.execute(*args, **cmd_options) File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 345, in execute output = self.handle(*args, **options) File "/usr/lib/python2.7/dist-packages/lava_server/management/commands/workers.py", line 77, in handle options["disabled"]) KeyError: 'disabled' iNote that before I have successfully done commands like adding user or device-types Regards

8 years

3
4
0 0

Device Integration Stories - request for contributions

by Neil Williams

Possibly the hardest thing that can be done with LAVA is to integrate a new device-type. Every story is different, every device has it's own issues. The LAVA software team would like to build a history of integration stories to document the recovery process, provide integration hints for others, build a wider picture of the requirements of automation across the ecosystem. A key part of any device integration is how to recover a device which has been taken offline by a broken build or broken test action. It is rare that a new device is suitable for LAVA directly out of the box, there are usually firmware updates, configuration changes, jumper settings or other mods required. Each of these steps needs to be documented so that someone else can integrate the same board into their instance and the same steps are useful when recovering a bricked device. Having the Jinja2 template is not usually enough. (As a side note on that, we do welcome all contributions of new jinja2 templates. The one place where everyone looks for new device-type support is lava_scheduler_app/tests/device-types/ in the master branch of http://git.linaro.org/lava/lava-server.git. If you have integrated a device-type which does not exist in that directory, please follow the contribution guidelines in the documentation and submit your Jinja2 template for review.) https://git.linaro.org/lava/lava-server.git/tree/lava_scheduler_app/tests/d… Recent device integrations in LAVA are documented through a process of comments on JIRA stories (all LAVA JIRA stories are accessible anonymously). Sometimes this turns out to be trial and error, so sometimes a README document or similar is created for easy reference. I'll post some of our recovery README documents on this thread later. This is a request for contributions of your integration stories, your device recovery README documents and your input on what are the minimum requirements for automation in general. It doesn't matter how unusual the hardware or how commonplace. It doesn't matter if it took months to integrate or days. Someone may need your information to setup the same device in their instance. If your device is available on the open market, someone could be thinking of how to automate it. Please post your recovery README texts, integration stories here. This list is publicly archived, so this should make it easy for others to find your contribution in their favourite search engine. In future, as well as unit tests, the LAVA software team may ask for the recovery notes and integration story to be posted here when reviewing contributions of Jinja2 templates. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years

1
0
0 0

Setting the "inactivity timeout" parameter

by Dragos Iorgulescu

Hello! Is there a way to set/override/disable the inactivity timeout? While working with a (sometimes) slower board, we end up with "Terminal.stop: Inactivity timeout." . I believe this might have to do with the fact that the board fails to respond in a timely fashion and LAVA marks this as hung. While searching for this in the docs, I only found a reference for hacking sessions. Kind regards, Dragos

8 years

2
2
0 0

rootfs resize in LAVA2

by Dragos Iorgulescu

Hi everyone! Does LAVA offer a way to resize rootfs images (“ramdisk_files”, as they are referred to in the code) before deploying them? We are working with our own Linux distro on a Cavium board and we need to resize the rootfs image (which is ext4) after the test overlays are added by LAVA. This step is necessary since we need to run a series of community tests that imply a large number of package dependencies. Also, we do not want to resize the whole image by default, since this will have other CI storage side effects and, also, some policy implications (that I am not really aware of 😊). Anyway, we are currently looking at adding/altering apply_overlay.py (pipeline/actions/) in order to run a resize2fs command after the “updated” ramdisk is unmounted. Just wondering if there is already a more elegant way of doing this. We are also considering making some extensions and maybe get to the point where we could specify this type of requirement in the device dictionary or the device template itself, and then specific code would be executed only if such a setting would be encountered at runtime, but that remains to be discussed. Thanks in advance! Dragoș

8 years

2
2
0 0

Local lava test

by Denis HUMEAU

Dear all, I'm quite sure I've seen somewhere in Lava documentation an article explaining how to replay some Lava jobs in a local mode, but I can't find this piece of information anymore. By local mode, I mean: - DUT at the developer desk, not in the test farm - Jobs available on a Lava server - Developer wants to replay/modify locally a job Can you share any link or help on this? Many thanks, Denis

8 years

5
5
0 0

Customizing the deployment and booting actions

by Ros Dos Santos, Alfonso (CT RDA DS EVO OPS DIA SE 1)

Hello everyone, in our current project we have some devices that are not "directly" supported by lava. I would like to ask for you opinion on which would be the correct way to proceed. The main problem is that our devices have a EFI implementation in the firmware that is making the task of installing uboot very hard. To avoid this, we thought about serving the image through an "emulated" usb stick. On that regard, we made some progress by setting up a secondary device that would use the linux g_mass_storage module to serve the image through the USB otg port. Our setup is then: 1) The actual testing device 2) The secondary device which only serves the image with g_mass_storage 3) The host machine running the lava-slave application. We thought that we could add a new device-type template to the lava server that would somehow override the deployment and boot actions to address our setup. We tried to look into the base.jinja2 file for some sort of entry point that would allow us to, for example, run a script that would first send the image to the secondary device and secondly run the g_mass_storage module with the image file. Unfortunately, it is still not clear for us how exactly we could inject/override those steps and ignore for instance the boot loader commands. The first question would be, are we going in the right direction? Is there any piece of documentation that would describe a similar problem? What would be your recommendations on this topic? thanks a lot in advance. Best, Alfonso

8 years

3
4
0 0

how to add the overlay to my os, start the test definition

by Xu, Hongyu

Hi , I deploy the device with grub(uefi) method. Now it can interrupt to grub>, and run the boot cmd and auto login. I use the pxe iso installation boot or the sata boot, not the nfs boot. But the lava-overlay file only add into ramdisk(initrd.img), it will disappear when os bootup. So I use the transfer_overlay in my job , but it do nothing, the test shell can't be find . Please give me some help! Job.yaml: actions: - deploy: to: tftp kernel: url: http://swlab004/1680141/vmlinuz-4.11.0-7.1.hxt.aarch64 type: zimage ramdisk: url: http://swlab004/1680141/initramfs-4.11.0-7.1.hxt.aarch64.img compression: gz install_overlay: false preseed: url: http://swlab004/centos7/anaconda-ks_1026.cfg os: centos timeout: minutes: 5 - boot: timeout: minutes: 200 connection: serial method: uefi-menu commands: ramdisk_boot auto_login: login_prompt: ".* login:" username: root password_prompt: 'Password:' password: root transfer_overlay: download_command: ifconfig;wget -S --progress=dot:giga unpack_command: tar -C / -xzf prompts: - 'Last login: .*' - '[root@.* .*]#' parameters: shutdown-message: "reboot: Restarting system" - test: timeout: minutes: 50 definitions: - repository: metadata: format: Lava-Test Test Definition 1.0 name: smoke-tests-basic description: "Basic system test command for Linaro Ubuntu images. The test runs basic commands like pwd, uname, vmstat, ifconfig, lscpu, lsusb and lsb_release." maintainer: - hongyu.xu(a)hxt-semitech.com os: - centos scope: - functional run: steps: - echo "test1a:" "pass" from: inline name: smoke-tests1 path: inline/smoke-tests.yaml Best Regards XuHongyu This email is intended only for the named addressee. It may contain information that is confidential/private, legally privileged, or copyright-protected, and you should handle it accordingly. If you are not the intended recipient, you do not have legal rights to retain, copy, or distribute this email or its contents, and should promptly delete the email and all electronic copies in your system; do not retain copies in any media. If you have received this email in error, please notify the sender promptly. Thank you.

8 years

2
5
0 0

LAVA and Mender

by Arnstein Kleven

Greetings, my embedded system is using Mender for over-the-air updates. The way Mender works is that the embedded system needs to have to rootfs partitions, where one is actively being used, and the other is written to when a over-the-air update is initiated. After the new rootfs is written, Mender will change a variable in the bootloader environment and reboot, so that the system boots into the new rootfs. If there is a problem booting into the new rootfs, the bootloader will be set to boot into the old one again. Has anyone here written LAVA tests for systems that use Mender? If so, have you experienced that Mender has complicated the test automation? Any Mender boot test templates would also be appreciated. -- Kind regards, Arnstein Kleven

8 years

2
2
0 0

how to auto login the os with lava

by Xu, Hongyu

Hi dear all, I boot my device with nfs in lava, but it can't auto login the os , I put the logs to attachment ! Please give me some idear. And I can boot and auto login successful with sata boot, use the same job.yaml file! My job 454 : device_type: amberwing_rep1 job_name: amberwing_rep_nfs priority: medium visibility: public metadata: docs-filename: amberwing_rep_nfs timeouts: job: minutes: 300 action: minutes: 200 connection: minutes: 100 actions: - deploy: to: tftp kernel: url: http://swlab004/centos7/iso20171026-0/images/pxeboot/vmlinuz type: zimage ramdisk: url: http://swlab004/centos7/iso20171026-0/images/pxeboot/initrd.img compression: xz install_overlay: false nfsrootfs: url: http://autotest002/tmp/rootfs_centos.tar.gz install_overlay: true preseed: url: http://swlab004/centos7/anaconda-ks_1026.cfg os: centos timeout: minutes: 5 - boot: timeout: minutes: 200 connection: serial method: uefi-menu auto_login: login_prompt: '.* login:' username: root password_prompt: 'Password:' password: root commands: nfs_boot prompts: - 'Last login: .*' - '[root@.* .*]#' parameters: shutdown-message: "reboot: Restarting system" - test: timeout: minutes: 50 definitions: - repository: metadata: format: Lava-Test Test Definition 1.0 name: smoke-tests-basic description: "Basic system test command for Linaro Ubuntu images. The test runs basic commands like pwd, uname, vmstat, ifconfig, lscpu, lsusb and lsb_release." maintainer: - hongyu.xu(a)hxt-semitech.com os: - centos scope: - functional devices: - amberwing_rep-022 run: steps: - echo "test1a:" "pass" from: inline name: smoke-tests1 path: inline/smoke-tests.yaml Best Regards XuHongyu This email is intended only for the named addressee. It may contain information that is confidential/private, legally privileged, or copyright-protected, and you should handle it accordingly. If you are not the intended recipient, you do not have legal rights to retain, copy, or distribute this email or its contents, and should promptly delete the email and all electronic copies in your system; do not retain copies in any media. If you have received this email in error, please notify the sender promptly. Thank you.

8 years

2
2
0 0

Migrating scheduling into lava-master

by Neil Williams

LAVA requires a range of singleton daemon processes to schedule, organise, publish and log the test job operations. 2017.12 will be retiring the lava-server daemon and the /var/log/lava-server/lava-scheduler.log, moving those operations into the lava-master daemon and the /var/log/lava-server/lava-master.log. This is one of the steps in the removal of V1. This leads a substantial change to how admins approach an instance running 2017.12 or later. When jobs don't appear to be running for some reason, the default action is to check /var/log/lava-server/lava-scheduler.log - once the instance is running 2017.12 or later, admins need to remember that it is correct for: * /var/log/lava-server/lava-scheduler.log to be empty * the status of the lava-server daemon to be active (exited) * the lava-server service to not restart At some point after 2017.12 has been running, admins can choose to archive or purge /var/log/lava-server/lava-scheduler.log* and / or disable the lava-server service using systemctl. The scheduler from 2017.12 onwards will consist of: * /var/log/lava-server/lava-master.log * /var/log/lava-server/lava-logs.log * the status of the lava-master daemon will be active (running) * the lava-master service will restart when admins request it. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years

1
0
0 0

Fitting the LAVA overlay into an Android rootfs

by Steve McIntyre

[ Related-ish to the discussion in the "Fastboot - Android boot image apply-overlay" thread, but also slightly different so I'm starting a new discussion here... ] Hi folks, We were just discussing in IRC about potential problems with Android images in LAVA, in terms of fitting the overlay. Ideally, the Android builds would have a rootfs configured to be as small as possible - it's automatically resized to fit the available space at first boot, and leaving space otherwise is inefficient. However, if we're going to add the LAVA test overlay into the rootfs, space needs to be left for it to fit. That's not *necessarily* a problem, except that at build time the Android builds don't know how much space we're going to need! This leads to a bit of a chicken-and-eff problem. Anibal has already hacked around this a little bit as a workaround in a job definition - see https://validation.linaro.org/scheduler/job/1652299/definition#defline92 In the lxc, he's converting and expanding the rootfs image to make more space, then converting back ready for flashing. This is still quite hacky - there's still no information available there about the size of the LAVA overlay. It might fit now, but what happens if there's a new LAVA version next month with (hypothetically) a much bigger overlay? So, I'm thinking about a better way to do things here, in 2 parts: 1 .At the point when we apply the overlay, we know roughly how big it is and we could do the resizing of the image to fit directly without needing this guesswork. It's a relatively easy thing to do. 2. [Optionally] If we end up growing the rootfs image ourselves, it *might* end up overflowing the size available when we come to flash things, but "fastboot flash" will check this and abort. I even suggested that (to fail more quickly) we could parse the GPT partition table for size ourselves, but Nicolas pointed out that not all jobs will necessarily have such a partition table. Instead, *maybe* we could include a "maximum rootfs size" parameter somewhere if we care. Otherwise, just rely on fastboot doing the right thing. Does this sound sensible? Cheers, -- Steve McIntyre steve.mcintyre(a)linaro.org <http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs

8 years

5
9
0 0

Progress on LAVA-1133

by Neil Williams

LAVA: base-uboot: should not always pass console= https://projects.linaro.org/browse/LAVA-1133 Changes have now been made in review https://review.linaro.org/22540 This is one of the changes which is being migrated from the old bugzilla to the lava-users mailing list, so there is no current thread on this list for this topic. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years

1
0
0 0

Fastboot - Android boot image apply-overlay

by Anibal Limon

Hi, I'm trying to apply-overlay (test tarball) to a ramdisk inside Android boot image , the image contains the kernel (zImage), ramdiisk (initrd.img) and a config file (bootimg.cfg). The Android boot image can be modified using abootimg, for example: $ abootimg -x boot.img $ abootimg-extract-unpack-initrd $ tar -xvzf overlay-5.2.4.tgz -C ramdisk $ abootimg-extract-pack-initrd $ abootimg -u boot.img -r init.rd Currently i can execute a test shell into LXC to make the extraction of the boot image [1][2], the issue i have is that the test overlay tarball (for example: overlay-5.2.4.tgz), isn't available because the creation is handled after the LXC pipeline by deploy action to the target device [3][4]. Any feedback would be appreciate. Best regards, Anibal [1] https://staging.validation.linaro.org/scheduler/job/196031/definition#defli… [2] https://staging.validation.linaro.org/scheduler/job/196031 [3] https://staging.validation.linaro.org/scheduler/job/196031/definition#defli… [4] https://git.linaro.org/lava/lava-dispatcher.git/tree/lava_dispatcher/action…

8 years, 1 month

2
3
0 0

error while executing LAVA power on script

by Philippe BEGNIC

Hello Lava team, We have implemented new scripts to support flashing in our "power on" script. This script manage the PDU, boot pins configuration of the board and an external ST flasher. At the beginning of the job, the "power on" script is launched. Then I got the following error message <line way too long ...> . This cause the execution script to be interrupted, but the job is not interrupted and can continue without flashing the board. I do not have any additional logs regarding this error on the LAVA web interface. Our power on script is composed of several scripts that are "chained", script power on call another script, that call another one. Do you have an idea about this error ? Details concerning our configuration : * we use LAVA V2 ( version 06/2017 ). * our jobs are pipelined jobs ( LAVA V2) * Lava server // dispatcher are installed through dockers. BR Philippe

8 years, 1 month

2
1
0 0

Moving to systemd

by Neil Williams

LAVA, until now, has packaged SysVinit files to manage the various daemons but there have been longstanding problems with log rotation and the need to carry redundant code to manage the daemonize process for python scripts. This has allowed instances to support both SysVinit and systemd because systemd automatically converts the SysVinit files to systemd service files. Now that V1 has been removed, it has become possible and advantageous to drop SysVinit support and move to only systemd support. This is expected to mean that lava-dispatcher (and therefore lava-server) will depend on systemd-sysv, i.e. systemd needs to be the init system to be able to start the daemons upon which LAVA relies. A default install of Debian Jessie and Stretch already defaults to using systemd as the init system, so this does not necessarily involve any changes. If there are questions or issues with this, please reply to the lava-users mailing list. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 1 month

1
0
0 0

Binary flashing test for Dragonboard 410

by Alfred Ziskind

Hi all, We are planning implementing a test that uses QDL (https://git.linaro.org/landing-teams/working/qualcomm/qdl.git/) for DragonBoard flashing. Scenario: after we flashed the binaries with QDL we want to check in the serial log if the boot sequence is correct. Can we develop this test in LAVA? If not, please recommend a test framework that can. Thank you, Alfred

8 years, 1 month

2
1
0 0

LAVA 2017.11 release

by Neil Williams

See https://lists.linaro.org/pipermail/lava-announce/2017-September/000037.html 2017.11 is the second release on the roadmap to the removal of V1. This is the single largest change ever made to the LAVA packages. * All dashboard URLs are permanently disabled in lava-server. * All devices which were not enabled for V2 are now hidden and unusable. * All V1 code has been removed from lava-dispatcher. * All V1 documentation has been removed from lava-server-doc. 2017.11 disables all access to any V1 test data - including V1 only devices. The data itself remains in the database but is not accessible via any API. If you have chosen to keep an archive of V1 test data and have not yet done so, you should do this before installing this upgrade. At the very least, you **must** have a backup of your postgresql database **before** you upgraded to 2017.11 in order to make a V1 test data archive later. 2017.11 requires that **all** workers and each associated master are upgraded to 2017.11 for test jobs to continue running. This is because there have been very large changes to the directories and file layout within both lava-server and lava-dispatcher associated with the removal of the V1 components. The LAVA software team have been checking ahead for the final stage of the roadmap and plans are on track, as described. http:// and https:// changes in django ====================================== A reminder, if you are accessing an instance using http:// (whether localhost or not), then the changes described in the documentation **must** be applied for the instance to operate correctly. This is a security change in Django, not a change in LAVA. https://staging.validation.linaro.org/static/docs/v2/pipeline-debug.html#us… Symptoms include HTTP500 errors but can also include the inability to login to an instance using LDAP or local django accounts, depending on your local apache configuration. Changes in 2017.11 ================== Changes in the 2017.11 release are described below, including the short git commit hash. Use the git commit hash to go directly to that commit using a URL like https://git.linaro.org/lava/lava-server.git/commit/?id=b19b9648a for lava-server or https://git.linaro.org/lava/lava-dispatcher.git/commit/?id=1451e3a4 for lava-dispatcher. Where the changelog message includes a story id like LAVA-552, that id can be added to the base URL: https://projects.linaro.org/browse/ to find out more about the change itself. e.g. https://projects.linaro.org/browse/LAVA-552 All LAVA stories are publicly available without requiring a login. 2017.11 also includes large changes to the packaging of both lava-server and lava-dispatcher. The prompts formerly used to configure a V1 remote worker have been removed. Packaging is maintained at https://github.com/Linaro/pkg-lava-server and https://github.com/Linaro/pkg-lava-dispatcher Packages for 2017.11 have been uploaded to Debian unstable and also the LAVA repositories (for jessie-backports and stretch-backports). lava-server =========== a799fc6c Fix links back to the main instance 5b9656b6 Fix broken plural in header b21f8c8f Fix crash in job_section_log c9526f01 Document the LOG_SIZE_LIMIT setting in settings.conf 63b57c94 Remove lava-mount-masterfs 75b403f4 Remove add_device script 8ddea823 Add description about the disc space requirement for pg_upgradecluster. 5c9dceb2 Remove TemporaryDevice imports 9b21ea48 Removed unused Worker functions 2accb4c3 Matching change to templates for directory change 1d384c4a Update docs regarding device-dictionary lava-server manage command. 32472fac Port 77fd10e2 to scheduler.devices.get_dictionary 114bbfa5 LAVA-1093 - Add a management command to drop all materialized views 8bc9d83a Update dashboard XML-RPC API help for api_version bb090e96 LAVA-552 - add system.api_version support 77fd10e2 LAVA-1092 support passing a context to device_config de7ad3aa Remove dashboard_app views and static files. 4bc68531 Remove templates from dashboard_app. 1a618c27 Remove unused functions d50deca7 api: limit the number of jobs returned 59880e14 Fix HTTP 500 on devicetype health history a39fecf5 Remove unused resources (css, js and images) a32ac3b6 Rename base-bootstrap and content-bootstrap 7c108de1 job: print lava.job result in case of failure b51c419f Remove old templates a9139c1f Use bootstrap template 7915f18f Remove unused resources 7331901a scheduler.api.jobs.show: add failure_comment 71233bff Fix print syntax for python3 compatibility 936d5eca Remove unused tags 4ebb2a39 Remove unused sources 21a7eb18 Save more sql queries by caching values 4d0d867e Remove non pipeline jobs support a6648dbd Retain docs on disabling V1 workers 8eac0d74 Revert "LAVA-950 set the master identity" 68573b1a Remove V1 dependencies a8151559 timing: handle new start/end line format 26cc3ad0 Remove unnecessary js lib beautify. c44009fa Simplify job view e1da85d4 Fix loading of description.yaml 7b41f7ef Add documentation for secondary media writer parameters 1b4d3dd5 LAVA-1081 migrate instance name to settings f3dc1987 Fix up typos from earlier changes faa24f59 LAVA-1079 - remove HIDE_V1 and HIDE_V2 doc options 942b84c3 Remove unused functions and imports e61a4831 LAVA-1056 Drop V1 documentation 126dce0c Fix crash introduced by 42451c9cc f0b28214 Remove v1 codebase 31bb1bf0 Fix notification exception when query is used for data comparison. a6cff700 Remove last references to lava_dispatcher v1 043690d9 Remove references to lava_dispatcher v1 4c9e493c Remove dashboard_app traces from rest of the codebase. c371b8ca drop link to removed page 1b1b638e LAVA-876 - Return empty data stubs for dashboard API calls. 919c10c2 Remove links to dashboard_app in the scheduler_app 1483e9fb Remove leftover from job wizard f7a1c976 docker: allow passing extra options 42451c9c Use all_jobs_with_custom_sort when applicable cc4ffd64 Fix broken links in example test job 0ca5d75c Drop the migration status page. b37fe944 Add a device-type config for the rpi-3b in armv7 32bit mode 9e904a18 Revert "LAVA-876 - Remove access to Dashboard" f07e0950 Fix warning regarding naive datetime e8348425 Remove more references to dashboard_app 103a44be LAVA-876 - Remove access to Dashboard d33c9c98 LAVA-876 - Remove access to Dashboard b6fc46aa Revert "LAVA-1038 add a settings to archive the instance" f75a333d Do not run dashboard test anymore d9163445 Remove references to v1 jobs addb61d9 bug 3268: fix lava-master crash with invalid logs febf91b1 Improve advice on api/help ac8b7df3 fix pep8 error 64025cf3 LAVA-1072 test all template connection syntax 10d5fbe1 Adjust U-Boot load addresses for tegra124 devices to allow bigger kernels f30ef936 LAVA-1065 - Remove dashboard_app urls server wide. 096a2f1c LAVA-1049 - Allow for .yml as well as .yaml for healthchecks 918f0bc7 Fix logic so addldapuser and mergeldapuser work for all LDAP configs 36fcb3c2 Documentation changes for multiple uarts b1de8acf Update instructions for migrating postgres 0ff08373 LAVA-1053 Results limit does not work for queries f9e64aa1 Fix SQL request storm when listing jobs 2d9c4757 No need to save after get_or_create or Create() 8988cb36 LAVA-896 fix level in result export 219fa3dd LAVA-1073 - support for a second UART on a device c934bbbe master: fix database reconnection 4f0e0e62 LAVA-950 set the master identity 316d77fb Expand the device integration guidance. lava-dispatcher =============== d91a5b99 Revert "Stop parsing kernel messages when the end of a panic has been found" 1451e3a4 LAVA-1098 unassigned variable in read_feedback 437d57ba Rationalise the devices directory structure bdffb3bf Add power control to pyocd d9553680 Add feedback check to test shell 06588ddf Receive the udev device directly from the udev rule d811b2a8 Fix license 65d6c6a6 Move arduino101 dfu example urls to images.v.l.o 61cff41c Only log when feedback content was found 0ba35392 Move juno removable URLs to images.validation.linaro.org cb983d97 Cleanup after pytest runs 997265c2 LAVA-1086 add handler to listen to feedback 29a60e38 Add ConfigObj dependency for TFTP check 70bd3fab Fix prospector warnings 6edb8ef9 Fix missing import and other pylint issues 6f7f5f05 Remove lava-lxc-device-* scripts which are unused. 578b054b Use py.test 65c03eca docker: allow passing extra options 7b7d920a device-info should be yaml ef8f7d4c Remove deprecated function 165e14f8 Fix linger period: set it to 10s instead of 5ms 49c500ac Fix double crash when the process crash early 7dbdc2b6 Add test case for secondary deployment writer tool parameters 969459c6 LAVA-1069 - migrate dispatcher device configs 9e8e5258 Use optargs to extend ci-run for python3 e1c1561b Fix python3 breakage in unit test 69277753 Read feedback when finalising connections 0fb44088 LAVA-1067 Refactor fastboot flash operations 9ac71872 Add missing import 94430958 Ensure finalize always finalises protocols 9d712364 Add option to not uniquify secondary image download paths cdfd5909 Fix master cert handling for lava-run and lxc helper 73548f7a Rename dd params to tool for removable media 1802c3e8 Handle master and slave certs as filenames only ed306477 Do not rely on SysV runlevel to ensure container is ready. 75b47c6a Remove loop mounts via guestfs, while applying overlay to sparse image. cd1152e3 Support archlinux and slackware rootfs 6fc024a5 Remove deprecated modules ddf5d2fc Remove v1 code 5561be23 LAVA-1074 - second UART support 758a41a6 Allow configurable writes for secondary storage deployment fb75db4a Add flag to disable uniquifying the download paths -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 1 month

1
0
0 0

403: Forbidden - CSRF verification failed (2017.10-1)

by Dragos Iorgulescu

Hi everyone! I have recently upgraded our production server to 2017.10-1. Whenever anyone tries to login in the WEB UI, either the “403: Forbidden – CSRF verification failed” error is returned, or no error at all (for the latter, the default LAVA page is shown again, but with no login info). I also tried creating a new superuser from the CLI. That worked, but I can’t use it to logon via the WEB UI, so I guess this has nothing to do with the user accounts stored in LAVA. Is this triggered by a change in Django/LAVA? Kind regards, Dragoș

8 years, 1 month

2
2
0 0

[lava v2] charts and redefining X attribute

by Jonathan Guyot

Hello everybody, I am working with both Jenkins and lava and I thought it was possible to change X attributes in charts to use Jenkins build number instead of a date. In my job, I defined a metadata like this : job_name: quad-ssh-host device_type: ssh *metadata: {jenkins-build: 725, job-context: nightly}* timeouts: job: minutes: 10 action: minutes: 3 connection: seconds: 20 visibility: public […] and I made a query that can select these jobs using job-context as a filter. Then, in the chart, I wanted to use the jenkins-build value to group job instead of a simple date, but the chart does not seem to be able to do that. I made many tries on the lava 2017.2 version. I wonder whether I am misusing lava charts or if it is an issue that need a fix. Thanks in advance for any reply. Jonathan

8 years, 1 month

2
5
0 0

”502 Proxy Error” for large job log & details (in the WEB UI)

by Dragos Iorgulescu

Hello again! After managing to run a consistent series of tests on one of our boards, I came across a situation where the job log seems to be too big for LAVA to display. When clicking the ”Job details” button, the server eventually sends a ”502 Proxy Error” message. Is there a setting or a workaround to make LAVA serve the job log & details in ”chunks”? I did download the plain log (using, for example, http://LAVA_HOST/scheduler/job/JOB_ID/log_file/plain), and it is, indeed, huge (29 MB), but I was just curious if, maybe a ”log rotation”-like mechanism could be used in these situations (somehow). Thanks in advance! -- /Dragoş

8 years, 1 month

3
4
0 0

Git error after upgrading to 2017.10-1: "fatal: index-pack failed"

by Dragos Iorgulescu

Hi! Just upgraded from 2017.6 to 2017.10-1. Tried re-running some jobs that worked fine just before the upgrade and noticed that git checkouts can no longer pe performed. The encountered error is: fatal: The remote end hung up unexpectedly\nfatal: early EOF\nfatal: index-pack failed\n Tried checking out manualy on the host, using the same repo. It works. How could I proceed investigating this? Could this be related to some lava user configuration that got reset (I don't know if git-related configs are associated with LAVA)? Thanks in advance (again)! P.S: I have attached the plain job log -- /Dragoş

8 years, 1 month

2
3
0 0

LAB Notice: Planned maintenance

by Dave Pigott

[bcc’d to everyone to ensure we get full coverage. Please ignore if you have no interest in LAVA and the Lab] Hi all, Starting this week, we are finalising the transition of LAVA main production to LAVA V2. The main update and hopefully minor downtime, will commence at 09:00 UTC on Monday 6th November. Along with this we will also be upgrading all micro-instances to Debian Stretch and LAVA 2017.11. There will be a brief period on Monday morning, which should be less than an hour, when remote logins will be disabled as we upgrade the gateway server. Please note, that when validation.linaro.org <http://validation.linaro.org/> comes back online, LAVA V1 job submissions will be rejected, so if you have any bots which auto-submit V1 jobs, please work with us to transition them to V2. For future reference, for announcements about Lab availability, you should join the lava-users mailing list. Thanks Dave ---------------- Dave Pigott LAVA Lab Lead Linaro Ltd t: (+44) (0) 1223 400063

8 years, 1 month

1
1
0 0

Very long output line seems to interfere with board output/job run

by Dragos Iorgulescu

Good morning everyone! While running some tests using our Linux distro and a SuperMicro Intel D1521 board, we encounter a situation where a very long line (shown as a "broken line" in LAVA's logs - in the WEB UI) seems to interfere with the job run, in a sense that the job eventually ends in a timeout. This does not happen when that particular set of tests is not run and I am wondering if this might be a LAVA issue. -------------------- A bit of context, first. SuperMicro boards "make sure" your board connection options are limited to using SoL (serial-over-lan), IPMI and a serial port (but in a very limited manner). One is not able to do a "typical" image load, since you may not specify hardware addresses or properly interact with the bootloader. This led to an approach which is best illustrated in the attached diagram. For short: LAVA instantiates an LXC container, from where we run a series of scripts to prepare the board (copy some images & files on an already available Linux instance, altering the Grub menu and then rebooting the board, making it start the image we wish to test) and then run remote commands via SSH. So, basically, the LXC container runs each and every command on the board, acting as the middleman. ------------------- Now, we have the following: * https://paste.debian.net/993516/ -> plain log excerpt that contains the very long line I was talking about (please see line no. 90) * https://paste.debian.net/993517/ -> job definition * https://paste.debian.net/993518/ -> test definition Is there a way to further investigate if this type of output is "giving LAVA a hard time"? Is there a setting that limits interpreted output, somehow? May it be that this is a LAVA issue? Kind regards! -- /Dragoş

8 years, 1 month

2
4
0 0

"No such method: u'results.run_query'" when trying to access queries from a Python script

by Dragos Iorgulescu

Good morning everyone! I am trying to "play around" with user queries in LAVA2. After reading about them (https://validation.linaro.org/static/docs/v2/lava-queries-charts.html?highl… ), I downloaded the provided sample script (https://validation.linaro.org/static/docs/v2/examples/source/query-results.… ), made the required updates (user names, query name, hostname & the such) and gave it a try. The only thing I get stuck on is an XML-RPC lib error: "No such method: u'results.run_query'" (the whole traceback can be seen here: https://paste.debian.net/993833/ ) I have attached the modified script I am trying to run at the moment and a screenshot of the query settings. The API Help page is not reachable/present at the moment (nor on https://validation.linaro.org/static/api/help or on our own LAVA instance ), so I am unable to check whether or not the method's name has changed since the last doc update. What am I missing or doing wrong? Many thanks in advance! Have a great day! -- /Dragoş

8 years, 1 month

2
4
0 0

PXE Booting and flashing device

by Rees, Kevron

I have a minnowboard turbot device that I want lava to boot, flash, reboot to flashed image, and then run tests. I would like to boot the device via PXE networking to an initramfs where I will flash an image to permanent storage. I can use efibootmgr within the initramfs image to set next-boot to the permanent storage device. Once booted from the permanent storage device, lava tests will run as per normal. It appears that many of the ipxe examples I have found simply run tests on the initramfs itself. Is it possible to flash the device once the initramfs has booted? It seems like it would be simple for lava-dispatcher to run wget http://someurl/foo.img | dd of=/dev/sda once the initramfs has reached login. However, I do not know if there is such a deploy mechanism. Does anything already exist within lava to help achieve this model? Is there a better way? Any feedback would be appreciated. -Kevron

8 years, 1 month

2
4
0 0

Truncated commands & output in LAVA2

by Dragos Iorgulescu

Hi all! We have been working on integrating a cn8304-based board (uses uboot). We noticed some strange behavior that might be related to LAVA2. On average, 2 out of 3 job runs end up with errors/timeouts. We use the same job definition and device each and every time. Looking through LAVA logs, it seems that commands and some output gets truncated, thus malforming the commands being sent or the prompts that LAVA expects. The prompt we expect is "EBB8304> ", but sometimes it gets truncated to "304>". Also, a command gets mixed up somehow, and instead of "setenv loadinitrd 'tftpboot 0x60000000 358/tftp-deploy-VLCRzw/ramdisk/ramdisk.ext4.gz.uboot'", the board receives "EBB8304> setenv loadkernel 'tftpboot 0x40setenv loadinitrd 'tftpboot 0x60000000 358/tftp-deploy-VLCRzw/ramdisk/ramdisk.ext4.gz.uboot'" Details: * The job definition: https://paste.debian.net/993349/ * An excerpt from the LAVA2 job log/output, showing what gets mixed up: https://paste.debian.net/993350/ * The plain job log is attached to this e-mail Many thanks in advance! -- /Dragoş

8 years, 1 month

2
2
0 0

LAB Notice: Planned maintenance

by Dave Pigott

Hi all, Starting this week, we are finalising the transition of LAVA main production to LAVA V2. The main update and hopefully minor downtime, will commence at 09:00 UTC on Monday 6th November. Along with this we will also be upgrading all micro-instances to Debian Stretch and LAVA 2017.11. There will be a brief period on Monday morning, which should be less than an hour, when remote logins will be disabled as we upgrade the gateway server. Please note, that when validation.linaro.org comes back online, LAVA V1 job submissions will be rejected, so if you have any bots which auto-submit V1 jobs, please work with us to transition them to V2. Thanks Dave ---------------- Dave Pigott LAVA Lab Lead Linaro Ltd t: (+44) (0) 1223 400063

8 years, 1 month

1
0
0 0

LAVA Error 503 Service Unavailable Error

by Tan, Jian Chern

Hey does anyone know why i'm getting an Error 503 Service Unavailable after installing LAVA on a new server? I have it installed on another system and i've never gotten this error on that system before. If it helps, i'm working behind a corporate proxy and here are the pertinent lines from lava-server.log in the apache log folder. [Fri Oct 06 20:01:57.729804 2017] [proxy:error] [pid 2887:tid 140627852187392] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8000 (127.0.0.1) failed [Fri Oct 06 20:01:57.729851 2017] [proxy_http:error] [pid 2887:tid 140627852187392] [client *IP*:56521] AH01114: HTTP: failed to make connection to backend: 127.0.0.1 Thanks! Jian Chern

8 years, 1 month

2
1
0 0

How to transfer a file from DUT to Server?

by Sathyamurthy, Raghuram

Hello, I working on a requirement for our project where I need to get some log from DUT and send back to Server. How can I achieve it? Thanks, Raghu

8 years, 1 month

2
1
0 0

lava-tool auth-add problem

by Michael Chen

Hi lava users, We're trying to use lava-tool to automate the job submit process, but having problem on using "lava-tool auth-add" command. The error message is ERROR: Username provided but no token found. I tried the instructions from the following page, but without luck. https://validation.linaro.org/static/docs/v2/lava-tool-issues.html I am pretty sure the user name, token, server url are correct since we tested it with the example python script from above page's second paragraph, and we can submit job through it. The version of lava-tool is 0.23-1 and lava-server is 2017.10-1+jessie. Does anyone have similar issue? Thank you. Michael Chen

8 years, 1 month

2
1
0 0

Test job failed due to uefi-menu-selector timeout

by Arthur She

Hi, I got a problem in LAVA v2. My test jobs failed due to uefi-menu-selector timeout occasionally. Like this one <https://staging.validation.linaro.org/scheduler/job/188583> and this one <https://staging.validation.linaro.org/scheduler/job/185376>. When it happened, I usually re-submit the test job and, sometimes, it got recovery and had the test job run successfully, like this <https://staging.validation.linaro.org/scheduler/job/185377>. However, we're working on our own CI loop now, the test job <https://staging.validation.linaro.org/scheduler/job/194340> is submitted automatically. It doesn't make sense to re-submit the test job. Any ideas? Thanks, Arthur

8 years, 1 month

3
2
0 0

Board At Desk v1.0, a downstream of LAVA and kernelci, just released.

by Agustin Benito Bethencourt

Hi, CIP just released Board At Desk v1.0 which is based on LAVAv2 and kernelci. Link: https://www.cip-project.org/blog/2017/10/18/cip-launches-bd-v1-0 From those involved in this release, thanks to all of you who make these technologies possible and also to those involved in kernelci.org project. Thank you also for your support these past months. Best Regards -- Agustin Benito Bethencourt Principal Consultant - FOSS at Codethink agustin.benito(a)codethink.co.uk

8 years, 1 month

1
0
0 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

lava-users