lava-users April 2021

lava-users@lists.lavasoftware.org

9 participants
14 discussions

by koti

Hi, My test job .yaml does not provide any error while health-check trigger from GUI and give error while submitting the same test job from the command line . you can find the command line submission error as mentioned below. " lavamaster@lava$lavacli -i admin@validation jobs submit /tmp/test.yaml Unable to submit /tmp/target.yaml: <Fault 400: "Problem with submitted job data: expected str for dictionary value @ data['actions'][0]['boot']['parameters']['shutdown-message']">" " I have reported same type of bug for "soft-reboot" command support and provided the patch by lava-team for this soft-reboot support issue i.e " https://git.lavasoftware.org/lava/lava/-/merge_requests/1067/diffs#05da71b3… " Can someone provide the path for "shutdown-message" support also? Regards, Koti

2 years

Unable to get kernel boot messages in to lava job while using lava framework & telnet

by Nagendra Singamsetti

Hi , I am trying to automate the linux boot functionality on my custom boards using LAVA framework. Previously I have done it on Renesas board using LAVA it works fine.I got linux boot messages on lava job whereas i am not getting with customised board. *Issue: *I am using trace32 debugger to flash images onto the board (working fine). After that i am calling minimal method in pipeline job to get prompt strings/login But i am not getting the linux boot messages on the job. always stucks at the following: [common] connect-device Connecting to device using 'telnet 10.10.0.18 4000' <http://10.10.0.69/scheduler/job/789#L136>Setting prompt string to ['lava-test: # '] <http://10.10.0.69/scheduler/job/789#L137>end: 2.3 connect-device (duration 00:00:00) [common] <http://10.10.0.69/scheduler/job/789#L138>end: 2 boot-trace32-image (duration 00:00:28) [common] <http://10.10.0.69/scheduler/job/789#action_3>start: 3 minimal-boot (timeout 00:03:00) [common] <http://10.10.0.69/scheduler/job/789#action_3-1>start: 3.1 connect-device (timeout 00:03:00) [common] <http://10.10.0.69/scheduler/job/789#L141>Already connected <http://10.10.0.69/scheduler/job/789#L142>end: 3.1 connect-device (duration 00:00:00) [common] <http://10.10.0.69/scheduler/job/789#action_3-2>start: 3.2 auto-login-action (timeout 00:03:00) [common] <http://10.10.0.69/scheduler/job/789#L144>Setting prompt string to ['Booting Linux'] <http://10.10.0.69/scheduler/job/789#L145>auto-login-action: Wait for prompt ['Booting Linux'] (timeout 00:03:00) <http://10.10.0.69/scheduler/job/789#L146>Trying 10.10.0.18... <http://10.10.0.69/scheduler/job/789#L147>Connected to 10.10.0.18. <http://10.10.0.69/scheduler/job/789#L148>Escape character is '^]'. <http://10.10.0.69/scheduler/job/789#L149> <http://10.10.0.69/scheduler/job/789#L150>alif_asic01 4000 [115200 N81] <http://10.10.0.69/scheduler/job/789#L151> <http://10.10.0.69/scheduler/job/789#L152>auto-login-action timed out after 180 seconds <http://10.10.0.69/scheduler/job/789#L153>end: 3.2 auto-login-action (duration 00:03:00) [common] *Both telnet & serial server(ser2net) is configured.* *Uart cable attached with serial server: usb-Profolic_Technology_Inc._Usb-serial_Controller-ifoo-port0* i am getting boot log messages using telnet manually but not with lava. *Manual log with telnet:* $ telnet 10.10.0.18 4000 Trying 10.10.0.18... Connected to 10.10.0.18. Escape character is '^]'. alif_asic01 4000 [115200 N81] Booting Linux on physical CPU 0x0 Linux version 5.4.25-00024-g9283a6810958-dirty (nishit@alif) (gcc version 9.2.0 (GCC)) #72 Tue Apr 20 15:41:31 IST 2021 #HGG1 CPU: ARMv8-a aarch32 Processor [411fd010] revision 0 (ARMv8), cr=50c5383d CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache #HGG1.1 OF: fdt: Machine model: corstone700 [PFA] yaml job thanks Regards Nagendra S

2 years, 10 months

Docker-compose setup broken by 2021.04

by Westermann, Oliver

Hey, By coincidence I setup a local test instance of the lava docker-compose setup. I usually use merge master regularly, so my .env file contained DC_SERVER_IMAGE=lavasoftware/lava-server:latest. When I started this up, I get > lava-publisher_1 | ModuleNotFoundError: No module named 'environ' in the logs from various containers based upon lava-server. I tested it on my production setup as well as tested fixing my revision to 2021.03.post1, which fixes this. I checked my changes to the setup and couldn't find anything that looks related, so maybe consider this a bug report? Full callstack below, best regards Oliver Westermann Full Error Log: lava-publisher_1 | Traceback (most recent call last): lava-publisher_1 | File "/usr/share/lava-server/postinst.py", line 326, in <module> lava-publisher_1 | sys.exit(main()) lava-publisher_1 | File "/usr/share/lava-server/postinst.py", line 263, in main lava-publisher_1 | config = load_configuration() lava-publisher_1 | File "/usr/share/lava-server/postinst.py", line 127, in load_configuration lava-publisher_1 | module = importlib.import_module("lava_server.settings.prod") lava-publisher_1 | File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module lava-publisher_1 | return _bootstrap._gcd_import(name[level:], package, level) lava-publisher_1 | File "<frozen importlib._bootstrap>", line 1006, in _gcd_import lava-publisher_1 | File "<frozen importlib._bootstrap>", line 983, in _find_and_load lava-publisher_1 | File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked lava-publisher_1 | File "<frozen importlib._bootstrap>", line 677, in _load_unlocked lava-publisher_1 | File "<frozen importlib._bootstrap_external>", line 728, in exec_module lava-publisher_1 | File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed lava-publisher_1 | File "/usr/lib/python3/dist-packages/lava_server/settings/prod.py", line 23, in <module> lava-publisher_1 | import environ lava-publisher_1 | ModuleNotFoundError: No module named 'environ'

2 years, 11 months

Unable to get kernel boot messages in to lava job while using lava framework & telnet

by koti

Hi Nagendra, I suspect the auto-login action functionality. Please try to test by avoiding auto-login functionality. We can nail down/avoid the auto-login functionality issue like below. Steps: ###### 1. Boot the target manually and login 2. Make sure you will get login prompt 3. Run reset/reboot command 4. Check whether we will get to see the console log in the LAVA result page. Best of luck! Note: 1. You can use below "-boot" section " - boot: timeout: minutes: 2 method: minimal soft_reboot: ["reset;pwd;reset"] parameters: shutdown-message: ["< Please mention shutdown message > "] boot-message: [" < Please mention boot message "] " Regards, Koti On Thu, 22 Apr 2021 at 21:33, <lava-users-request(a)lists.lavasoftware.org> wrote: > Send Lava-users mailing list submissions to > lava-users(a)lists.lavasoftware.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.lavasoftware.org/mailman/listinfo/lava-users > or, via email, send a message with subject or body 'help' to > lava-users-request(a)lists.lavasoftware.org > > You can reach the person managing the list at > lava-users-owner(a)lists.lavasoftware.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Lava-users digest..." > > > Today's Topics: > > 1. Unable to get kernel boot messages in to lava job while using > lava framework & telnet (Nagendra Singamsetti) > 2. Notify on finish does not hold duration (Westermann, Oliver) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 22 Apr 2021 17:53:31 +0530 > From: Nagendra Singamsetti <nag.singam91(a)gmail.com> > To: lava-users(a)lists.lavasoftware.org > Subject: [Lava-users] Unable to get kernel boot messages in to lava > job while using lava framework & telnet > Message-ID: > <CAFhg_Wste+2=zO51aA3GOq-BjwbUx76rms= > gF8Rv-hKNCOeQVQ(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi , > > I am trying to automate the linux boot functionality on my custom boards > using LAVA framework. Previously I have done it on Renesas board using LAVA > it works fine.I got linux boot messages on lava job whereas i am not > getting with customised board. > > *Issue: *I am using trace32 debugger to flash images onto the board > (working fine). After that i am calling minimal method in pipeline job to > get prompt strings/login > But i am not getting the linux boot messages on the job. always stucks at > the following: > [common] connect-device Connecting to device using 'telnet 10.10.0.18 4000' > <http://10.10.0.69/scheduler/job/789#L136>Setting prompt string to > ['lava-test: # '] <http://10.10.0.69/scheduler/job/789#L137>end: 2.3 > connect-device (duration 00:00:00) [common] > <http://10.10.0.69/scheduler/job/789#L138>end: 2 boot-trace32-image > (duration 00:00:28) [common] > <http://10.10.0.69/scheduler/job/789#action_3>start: > 3 minimal-boot (timeout 00:03:00) [common] > <http://10.10.0.69/scheduler/job/789#action_3-1>start: 3.1 connect-device > (timeout 00:03:00) [common] <http://10.10.0.69/scheduler/job/789#L141 > >Already > connected <http://10.10.0.69/scheduler/job/789#L142>end: 3.1 > connect-device > (duration 00:00:00) [common] > <http://10.10.0.69/scheduler/job/789#action_3-2>start: 3.2 > auto-login-action (timeout 00:03:00) [common] > <http://10.10.0.69/scheduler/job/789#L144>Setting prompt string to > ['Booting Linux'] <http://10.10.0.69/scheduler/job/789#L145 > >auto-login-action: > Wait for prompt ['Booting Linux'] (timeout 00:03:00) > <http://10.10.0.69/scheduler/job/789#L146>Trying 10.10.0.18... > <http://10.10.0.69/scheduler/job/789#L147>Connected to 10.10.0.18. > <http://10.10.0.69/scheduler/job/789#L148>Escape character is '^]'. > <http://10.10.0.69/scheduler/job/789#L149> > <http://10.10.0.69/scheduler/job/789#L150>alif_asic01 4000 [115200 N81] > <http://10.10.0.69/scheduler/job/789#L151> > <http://10.10.0.69/scheduler/job/789#L152>auto-login-action timed out > after > 180 seconds <http://10.10.0.69/scheduler/job/789#L153>end: 3.2 > auto-login-action (duration 00:03:00) [common] > > *Both telnet & serial server(ser2net) is configured.* > *Uart cable attached with serial server: > usb-Profolic_Technology_Inc._Usb-serial_Controller-ifoo-port0* > > i am getting boot log messages using telnet manually but not with lava. > > *Manual log with telnet:* > $ telnet 10.10.0.18 4000 > Trying 10.10.0.18... > Connected to 10.10.0.18. > Escape character is '^]'. > > alif_asic01 4000 [115200 N81] > > Booting Linux on physical CPU 0x0 > Linux version 5.4.25-00024-g9283a6810958-dirty (nishit@alif) (gcc version > 9.2.0 (GCC)) #72 Tue Apr 20 15:41:31 IST 2021 > #HGG1 > CPU: ARMv8-a aarch32 Processor [411fd010] revision 0 (ARMv8), cr=50c5383d > CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache > #HGG1.1 > OF: fdt: Machine model: corstone700 > > [PFA] yaml job > > thanks > > Regards > Nagendra S >

2 years, 11 months

Notify on finish does not hold duration

by Westermann, Oliver

Hey, I'm using the 'notify by mail' option for our setup. It's setup as criteria: status: finished But when the mail is send, it seems the the job is not really finished yet, as the mail do contain timestamps for submitted and started, but not for finished. This also results in Duration being None. Also I noticed that the "Device Details" and "Job Details" elements are empty for me. Is this "expected" behavior or is my setup wrong? (Using LAVA 2021.03.post1, but existed on 2021.1 as well) Best regards, Olli

2 years, 12 months

How /dev/log could be seen in lava docker slave?

by Larry Shen

Hi, guys, I have a question related to udev share: The logger of lava dispatcher host was configured as "/dev/log", which link to "/dev/log -> /run/systemd/journal/dev-log". How could I see them in lava docker slave? (Not native Debian slave). Thanks.

2 years, 12 months

Re: [Lava-users] LAVA master backup/restore

by Philippe Mazet (OSS)

Ok. You get 2 MRs for the price of one : In docker-compose tree: https://git.lavasoftware.org/lava/pkg/docker-compose/-/merge_requests/37 In LAVA documentation: https://git.lavasoftware.org/lava/lava/-/merge_requests/1479 May I ask an update about our other pending MRs? ;-) Thanks a lot in advance, Philippe From: Lava-users <lava-users-bounces(a)lists.lavasoftware.org> On Behalf Of Remi Duraffort Sent: Wednesday, April 14, 2021 9:20 AM To: Philippe Mazet (OSS) <philippe.mazet(a)oss.nxp.com> Cc: lava-users(a)lists.lavasoftware.org <lava-users(a)lavasoftware.org> Subject: Re: [Lava-users] LAVA master backup/restore Hello, if you have some time, a patch to update the documentation would be greatly appreciated !! Thanks a lot. Le ven. 2 avr. 2021 à 11:34, Philippe Mazet (OSS) <philippe.mazet(a)oss.nxp.com> a écrit : Hi Rémi, I finally found this to work: For creating the backup, run backup command as postgres user, eg: docker exec --user postgres docker-compose_db_1 bash -c "pg_dump --username=lavaserver lavaserver > /var/lib/postgresql/data/lavaserver.sql" Then for restoring backup, place the sql file in a folder that will be mounted as a volume in postgres container, on /docker-entrypoint-initdb.d/. On way to do this is to create a dedicated docker-compose file (docker-compose-restore-backup.yaml) that will be appended to docker-compose.yaml: version: "3.4" services: db: volumes: - ./initdb.d:/docker-entrypoint-initdb.d Then: docker-compose stop docker container rm docker-compose_db_1; docker volume rm lava-server-pgdata docker-compose -f docker-compose.yaml -f docker-compose-restore-backup.yaml up -d I hope that helps. Philippe From: Lava-users <lava-users-bounces(a)lists.lavasoftware.org> On Behalf Of Remi Duraffort Sent: Friday, March 26, 2021 4:56 PM To: Philippe Mazet (OSS) <philippe.mazet(a)oss.nxp.com> Cc: lava-users(a)lists.lavasoftware.org <lava-users(a)lavasoftware.org> Subject: Re: [Lava-users] LAVA master backup/restore Hello Philippe, I haven't tried so I don't have the detailed procedure. When you find out, could you share the procedure ? Thanks Le mar. 23 mars 2021 à 11:28, Philippe Mazet (OSS) <philippe.mazet(a)oss.nxp.com> a écrit : Thanks Rémi, but result is the same when I start only the db container, even after a full cleanup (docker container rm docker-compose_db_1 && docker volume rm lava-server-pgdata): ERROR: database "lavaserver" already exists Seems to be related somehow to "POSTGRES_USER: lavaserver" in docker-compose.yaml. Reading the entrypoint script for postgres (https://github.com/docker-library/postgres/blob/master/docker-entrypoint.sh), I thought using PGUSER could help, but I didn't manage yet to get that working. I will continue digging into this... but if you have a detailed procedure somewhere, please share 😊 Thanks a lot! Philippe From: Lava-users <lava-users-bounces(a)lists.lavasoftware.org> On Behalf Of Remi Duraffort Sent: Monday, March 22, 2021 9:25 AM To: Philippe Mazet (OSS) <philippe.mazet(a)oss.nxp.com> Cc: lava-users(a)lists.lavasoftware.org <lava-users(a)lavasoftware.org> Subject: Re: [Lava-users] LAVA master backup/restore Le jeu. 18 mars 2021 à 16:22, Philippe Mazet (OSS) <philippe.mazet(a)oss.nxp.com> a écrit : Hi all, I am using the docker-compose solution to run a master: https://git.lavasoftware.org/lava/pkg/docker-compose I was wondering how to extract a backup of the DB, and re-inject it in a different instance. I found that postgres image has its own "restore" mechanism, based on initdb.d folder, as mentioned in the documentation: https://registry.hub.docker.com/_/postgres/ The entrypoint.sh script (https://github.com/docker-library/postgres/blob/master/docker-entrypoint.sh) handles the restore whenever the folder is /docker-entrypoint-initdb.d/ contains a .sql file. But when we store a backup in the container's /docker-entrypoint-initdb.d/ folder, and remove both the postgres image and its db-data volume, we get this error on next start: ERROR: database "lavaserver" already exists Full startup log attached. Questions: - Does Linaro's docker-compose solution modify postgres restore mechanism in any way? I guess that's only because lava will automatically create an empty lavaserver database. So in your use case, you should only start the db service and not any lava-* services. When the restoration has been done, you can restart every lava services. - How do you handle backup/restore at linaro? Thanks a lot in advance, Philippe Mazet NXP Semiconductors - Edge Processing Email: philippe.mazet(a)nxp.com _______________________________________________ Lava-users mailing list Lava-users(a)lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users -- Rémi Duraffort LAVA Architect Linaro -- Rémi Duraffort LAVA Architect Linaro -- Rémi Duraffort TuxArchitect Linaro

3 years

LAVA exporting bash shell while using debian OS parameter

by P T, Sarath

Hi Team, I have used the LAVA job definition including deployment data(OS parameter) as debian . As per the deployment_data.py it should export the shell as /bin/bash but it's exporting /bin/sh. I am observing this thing after updating LAVA to 2021.01. Below I am giving the job definition file which I used. It will be very helpful if anyone reply on this. Thank you . Job definition ============= device_type: x86-simatic-slll job_name: x86-simatic-ipc227e-slll health-check timeouts: job: minutes: 20 action: minutes: 20 connection: minutes: 10 priority: medium visibility: public tags: - slll-simatic-ipc-227e-01 actions: - deploy: to: overlay - boot: method: minimal reset: true failure_retry: 2 auto_login: login_prompt: 'ebsy-isar login:' username: root password_prompt: 'Password:' password: root prompts: - root@ebsy-isar:~# transfer_overlay: download_command: wget unpack_command: tar -C / -xzf # TEST_BLOCK - test: timeout: minutes: 5 definitions: - repository: metadata: format: Lava-Test Test Definition 1.0 name: kernel-version-basic description: "check kernel version" os: - debian scope: - functional run: steps: - lava-test-case uname --shell uname -a from: inline name: kernel-version-inline path: inline/kernel-version-basic.yaml lava-signal: kmsg Regards Sarath P T

3 years

Re: [Lava-users] LAVA master backup/restore

by Philippe Mazet (OSS)

Hi Rémi, I finally found this to work: For creating the backup, run backup command as postgres user, eg: docker exec --user postgres docker-compose_db_1 bash -c "pg_dump --username=lavaserver lavaserver > /var/lib/postgresql/data/lavaserver.sql" Then for restoring backup, place the sql file in a folder that will be mounted as a volume in postgres container, on /docker-entrypoint-initdb.d/. On way to do this is to create a dedicated docker-compose file (docker-compose-restore-backup.yaml) that will be appended to docker-compose.yaml: version: "3.4" services: db: volumes: - ./initdb.d:/docker-entrypoint-initdb.d Then: docker-compose stop docker container rm docker-compose_db_1; docker volume rm lava-server-pgdata docker-compose -f docker-compose.yaml -f docker-compose-restore-backup.yaml up -d I hope that helps. Philippe From: Lava-users <lava-users-bounces(a)lists.lavasoftware.org> On Behalf Of Remi Duraffort Sent: Friday, March 26, 2021 4:56 PM To: Philippe Mazet (OSS) <philippe.mazet(a)oss.nxp.com> Cc: lava-users(a)lists.lavasoftware.org <lava-users(a)lavasoftware.org> Subject: Re: [Lava-users] LAVA master backup/restore Hello Philippe, I haven't tried so I don't have the detailed procedure. When you find out, could you share the procedure ? Thanks Le mar. 23 mars 2021 à 11:28, Philippe Mazet (OSS) <philippe.mazet(a)oss.nxp.com> a écrit : Thanks Rémi, but result is the same when I start only the db container, even after a full cleanup (docker container rm docker-compose_db_1 && docker volume rm lava-server-pgdata): ERROR: database "lavaserver" already exists Seems to be related somehow to "POSTGRES_USER: lavaserver" in docker-compose.yaml. Reading the entrypoint script for postgres (https://github.com/docker-library/postgres/blob/master/docker-entrypoint.sh), I thought using PGUSER could help, but I didn't manage yet to get that working. I will continue digging into this... but if you have a detailed procedure somewhere, please share 😊 Thanks a lot! Philippe From: Lava-users <lava-users-bounces(a)lists.lavasoftware.org> On Behalf Of Remi Duraffort Sent: Monday, March 22, 2021 9:25 AM To: Philippe Mazet (OSS) <philippe.mazet(a)oss.nxp.com> Cc: lava-users(a)lists.lavasoftware.org <lava-users(a)lavasoftware.org> Subject: Re: [Lava-users] LAVA master backup/restore Le jeu. 18 mars 2021 à 16:22, Philippe Mazet (OSS) <philippe.mazet(a)oss.nxp.com> a écrit : Hi all, I am using the docker-compose solution to run a master: https://git.lavasoftware.org/lava/pkg/docker-compose I was wondering how to extract a backup of the DB, and re-inject it in a different instance. I found that postgres image has its own "restore" mechanism, based on initdb.d folder, as mentioned in the documentation: https://registry.hub.docker.com/_/postgres/ The entrypoint.sh script (https://github.com/docker-library/postgres/blob/master/docker-entrypoint.sh) handles the restore whenever the folder is /docker-entrypoint-initdb.d/ contains a .sql file. But when we store a backup in the container's /docker-entrypoint-initdb.d/ folder, and remove both the postgres image and its db-data volume, we get this error on next start: ERROR: database "lavaserver" already exists Full startup log attached. Questions: - Does Linaro's docker-compose solution modify postgres restore mechanism in any way? I guess that's only because lava will automatically create an empty lavaserver database. So in your use case, you should only start the db service and not any lava-* services. When the restoration has been done, you can restart every lava services. - How do you handle backup/restore at linaro? Thanks a lot in advance, Philippe Mazet NXP Semiconductors - Edge Processing Email: philippe.mazet(a)nxp.com _______________________________________________ Lava-users mailing list Lava-users(a)lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users -- Rémi Duraffort LAVA Architect Linaro -- Rémi Duraffort LAVA Architect Linaro

3 years

Re: [Lava-users] Worker/master version mismatch

by Philippe Mazet (OSS)

I have submitted a MR to allow version mismatch: https://git.lavasoftware.org/lava/lava/-/merge_requests/1466 This feature is really a *must-have* for us. At our own risks of course. Thanks for considering it! We also have quite a few other MRs in the pipe. Some of them are few months old, even if all comments are replied. What is the appropriate way to get them reviewed/merged? ;-) Thanks a lot, Philippe From: Lava-users <lava-users-bounces(a)lists.lavasoftware.org> On Behalf Of Remi Duraffort Sent: Tuesday, January 26, 2021 10:04 AM To: Philippe Mazet (OSS) <philippe.mazet(a)oss.nxp.com> Cc: lava-users(a)lists.lavasoftware.org <lava-users(a)lavasoftware.org> Subject: Re: [Lava-users] Worker/master version mismatch Hello Philippe, I tried the lava-docker-worker but found it too restrictive, since we can’t customize the Docker container at all (adding some tools, cfg files, ssh keys, etc…), and was imposing usage of Docker within the job themselves (that would force us to rewrite many of our existing tests). That's an interesting use case. Maybe something that we might want to support. A possibility would be to specify the docker registry that you want to use instead of docker hub. As to “automatically run the right worker version”, I might have missed something. By checking lava-docker-worker sources, I concluded the Docker would simply refuse to start if version differs versus master. But do you mean it will automatically restart with the correct version when master is upgraded? That would be an easy fix (just adding a settings to lava_server and return depending on the setting value). But keep in mind that this might break at any release. Rgds -- Rémi Duraffort LAVA Architect Linaro

3 years

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

lava-users April 2021