Hi Lava Users,
After upgrading to LAVA 2021.01, we are facing an issue of jobs not completing/exiting (once submitted) and getting stuck in "Running" state, doesn't matter if the job is successful or unsuccessful/incomplete/fail. After sometime if we try to cancel the job it gets stuck in "Cancelling" state. Due to this the device gets blocked and subsequent jobs are sent to "Scheduled" state.
Now only possible solution to free up the device, is to delete the job from LAVA administration
LAVA Administration > Test jobs (under lava_scheduler_app) > {job id} > Delete .
Though this way the device is recovered but the whole job gets deleted.
I have tried with different boot and deploy methods like nfs, uuu, ums, and minimal. In all cases I am getting similar issue.
This issue is only observed while using imx devices.
Ssh based jobs are working properly and also other x86 based devices while booting through minimal method.
I have attached screenshot of the jobs (Lava-job-begin.PNG and Lava-job-end.PNG).
What can be the issue here ?
Thanks,
Bhargav
Thanks a lot for your reply!
The LAVA link you sent seems pretty outdated, and I don't think it can be applied as-is to docker-compose solution. It is for a host install on Debian (not Docker).
More recent documentation seems to be here: https://lava.readthedocs.io/en/latest/admin/basic-tutorials/instance/backup/
However it tells how to create a .sql backup file (which I have), not how to restore it.
About the postgres link you sent, it is for postgres running on host, not within docker I think.
For Docker, it is described in https://registry.hub.docker.com/_/postgres/ (chapter "Initialization scripts").
But then when I try to place my .sql file in /docker-entrypoint-initdb.d/ folder, I get the error mentioned in my email: database "lavaserver" already exists.
Thus my question: what procedure is followed by Linaro for the backup/restore of their docker-compose instances?
Thanks again,
Philippe
From: Lava-users <lava-users-bounces(a)lists.lavasoftware.org> On Behalf Of koti
Sent: Friday, March 19, 2021 2:15 PM
To: lava-users(a)lists.lavasoftware.org
Subject: [Lava-users] LAVA master backup/restore
Hi Philippe Mazet,
Please find my comments inline.
Questions:
- Does Linaro's docker-compose solution modify postgres restore mechanism in any way?
[Koti] --> I am not sure about this.
- How do you handle backup/restore at linaro?
[Koti] ----> still are you facing the backup/restore issue after following the below documentation?
1. https://docs.lavasoftware.org/lava/admin-backups.html
2. http://postgresguide.com/utilities/backup-restore.html
Regards,
Koti
On Fri, 19 Mar 2021 at 17:30, <mailto:lava-users-request@lists.lavasoftware.org> wrote:
Send Lava-users mailing list submissions to
lava-users(a)lists.lavasoftware.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.lavasoftware.org/mailman/listinfo/lava-users
or, via email, send a message with subject or body 'help' to
lava-users-request(a)lists.lavasoftware.org
You can reach the person managing the list at
lava-users-owner(a)lists.lavasoftware.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Lava-users digest..."
Today's Topics:
1. LAVA master backup/restore (Philippe Mazet (OSS))
----------------------------------------------------------------------
Message: 1
Date: Thu, 18 Mar 2021 15:22:20 +0000
From: "Philippe Mazet (OSS)" <mailto:philippe.mazet@oss.nxp.com>
To: "mailto:lava-users@lists.lavasoftware.org" <mailto:lava-users@lavasoftware.org>
Subject: [Lava-users] LAVA master backup/restore
Message-ID:
<PR3PR04MB7244B9C1456CAD918574F1218B699(a)PR3PR04MB7244.eurprd04.prod.outlook.com>
Content-Type: text/plain; charset="us-ascii"
Hi all,
I am using the docker-compose solution to run a master: https://git.lavasoftware.org/lava/pkg/docker-compose
I was wondering how to extract a backup of the DB, and re-inject it in a different instance.
I found that postgres image has its own "restore" mechanism, based on initdb.d folder, as mentioned in the documentation:
https://registry.hub.docker.com/_/postgres/
The entrypoint.sh script (https://github.com/docker-library/postgres/blob/master/docker-entrypoint.sh) handles the restore whenever the folder is /docker-entrypoint-initdb.d/ contains a .sql file.
But when we store a backup in the container's /docker-entrypoint-initdb.d/ folder, and remove both the postgres image and its db-data volume, we get this error on next start:
ERROR: database "lavaserver" already exists
Full startup log attached.
Questions:
- Does Linaro's docker-compose solution modify postgres restore mechanism in any way?
- How do you handle backup/restore at linaro?
Thanks a lot in advance,
Philippe Mazet
NXP Semiconductors - Edge Processing
Email: philippe.mazet(a)nxp.com
Hi Philippe Mazet,
Please find my comments inline.
Questions:
- Does Linaro's docker-compose solution modify postgres restore mechanism
in any way?
[Koti] --> I am not sure about this.
- How do you handle backup/restore at linaro?
[Koti] ----> still are you facing the backup/restore issue after
following the below documentation?
1. https://docs.lavasoftware.org/lava/admin-backups.html
2. http://postgresguide.com/utilities/backup-restore.html
Regards,
Koti
On Fri, 19 Mar 2021 at 17:30, <lava-users-request(a)lists.lavasoftware.org>
wrote:
> Send Lava-users mailing list submissions to
> lava-users(a)lists.lavasoftware.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.lavasoftware.org/mailman/listinfo/lava-users
> or, via email, send a message with subject or body 'help' to
> lava-users-request(a)lists.lavasoftware.org
>
> You can reach the person managing the list at
> lava-users-owner(a)lists.lavasoftware.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Lava-users digest..."
>
>
> Today's Topics:
>
> 1. LAVA master backup/restore (Philippe Mazet (OSS))
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 18 Mar 2021 15:22:20 +0000
> From: "Philippe Mazet (OSS)" <philippe.mazet(a)oss.nxp.com>
> To: "lava-users(a)lists.lavasoftware.org" <lava-users(a)lavasoftware.org>
> Subject: [Lava-users] LAVA master backup/restore
> Message-ID:
> <
> PR3PR04MB7244B9C1456CAD918574F1218B699(a)PR3PR04MB7244.eurprd04.prod.outlook.com
> >
>
> Content-Type: text/plain; charset="us-ascii"
>
> Hi all,
>
> I am using the docker-compose solution to run a master:
> https://git.lavasoftware.org/lava/pkg/docker-compose
> I was wondering how to extract a backup of the DB, and re-inject it in a
> different instance.
>
> I found that postgres image has its own "restore" mechanism, based on
> initdb.d folder, as mentioned in the documentation:
> https://registry.hub.docker.com/_/postgres/
>
> The entrypoint.sh script (
> https://github.com/docker-library/postgres/blob/master/docker-entrypoint.sh)
> handles the restore whenever the folder is /docker-entrypoint-initdb.d/
> contains a .sql file.
>
> But when we store a backup in the container's
> /docker-entrypoint-initdb.d/ folder, and remove both the postgres image and
> its db-data volume, we get this error on next start:
> ERROR: database "lavaserver" already exists
> Full startup log attached.
>
> Questions:
> - Does Linaro's docker-compose solution modify postgres restore mechanism
> in any way?
> - How do you handle backup/restore at linaro?
>
> Thanks a lot in advance,
>
>
> Philippe Mazet
> NXP Semiconductors - Edge Processing
> Email: philippe.mazet(a)nxp.com
>
>
>
>
Hello lava users,
I have to deal with a board with a very limited storage size. Overall capacity is 64 MB, with about 30 MB remaining.
This is a clear limitation for Lava usage. Hence my questions:
* Is there a possibility to configure Lava tests installation on an external storage, such as an USB drive?
* In Lava recent versions, is it possible to optimize the overlay tests structure?
* Today, assuming that tests are stored in a git repo, for each test the complete repo is copied in each overlay
* A balance has to be made between one git containing all the tests and one git per test (extreme configurations)
* Is there a better way to manage this? In a ideal world, that would mean copy only what is needed and not more
Thanks, have a nice day
Hey,
I'm hosting a lava-setup for us which is based upon the lava-docker-compose setup from here https://git.lavasoftware.org/lava/pkg/docker-compose
I managed to configure Django to use a EMAIL_HOST which allows me to notify my users. However dbutils.get_domain() returns example.com and some digging lead me to the site setup from Django. I'm not entirely sure how it's intended to work but I would expect that LAVA fetches the host to use from .env, where I specified DC_LAVA_SERVER_HOSTNAME=lava.mydomain.com.
Where would I set that or what did I miss?
Also:
I figured out a few details about the docker-compose setup by now, eg how to configure LDAP. Where would I add such documentation? A PR to the docs?
Best regards, Olli
Hello,
While testing 2020.10 with https enabled, we have noticed the following problem:
In /etc/lava-dispatcher/lava-worker, if we set WS_URL="--ws-url http://<master's address>/ws/" (or do not this variable at all), we get the following error on worker:
2020-11-19 15:14:57,751 INFO PING => server
SSL error in data received
protocol: <asyncio.sslproto.SSLProtocol object at 0x7f9a0d704160>
transport: <_SelectorSocketTransport fd=8 read=polling write=<idle, bufsize=0>>
Traceback (most recent call last):
File "/usr/lib/python3.7/asyncio/sslproto.py", line 526, in data_received
ssldata, appdata = self._sslpipe.feed_ssldata(data)
File "/usr/lib/python3.7/asyncio/sslproto.py", line 207, in feed_ssldata
self._sslobj.unwrap()
File "/usr/lib/python3.7/ssl.py", line 767, in unwrap
return self._sslobj.shutdown()
ssl.SSLError: [SSL: KRB5_S_INIT] application data after close notify (_ssl.c:2609)
Still, even with those errors the worker is fully functional.
Now, the only way to get rid of such errors is to explicitly specify the port, 8001:
WS_URL="--ws-url http://<master's address>:8001/ws/"
Have you experienced similar problems? Did we miss something on our setup?
Thanks a lot,
Philippe
