Im deploying lava using the docker container and I am looking at the correct paths to mount as volumes as to not lose data between cluster rollovers. (and to have to backups) If I change the instance.conf to point to a database that is outside of the container that gets backed up and includes these paths in my volume mounts is that all I need to do? Or is there additional paths/files that should be included.
https://master.lavasoftware.org/static/docs/v2/admin-backups.html
If anyone knows anything about this thanks!
Hello,
when trying to backup a lava instance, keep in mind that lava is made of two parts : the server (only one) and the dispatchers (usually more than one).
For the server you should backup: * the database (see /etc/lava-server/instance.conf) * the configuration: ** /etc/lava-server/ ** /etc/lava-dispatcher/certificates.d * the job artifacts: /var/lib/lava-server/default/media/job-output
On the dispatchers, you should backup: * dispatcher config: /etc/lava-dispatcher/ * all the changes you did like, /etc/ser2net.conf, udev rules, ...
Hope that helps.
Le jeu. 6 déc. 2018 à 18:35, Chris Sauer csauer@peoplenetonline.com a écrit :
Im deploying lava using the docker container and I am looking at the correct paths to mount as volumes as to not lose data between cluster rollovers. (and to have to backups) If I change the instance.conf to point to a database that is outside of the container that gets backed up and includes these paths in my volume mounts is that all I need to do? Or is there additional paths/files that should be included.
https://master.lavasoftware.org/static/docs/v2/admin-backups.html
If anyone knows anything about this thanks!
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
On Fri, 7 Dec 2018 at 12:21, Remi Duraffort remi.duraffort@linaro.org wrote:
Hello,
when trying to backup a lava instance, keep in mind that lava is made of two parts : the server (only one) and the dispatchers (usually more than one).
For the server you should backup:
- the database (see /etc/lava-server/instance.conf)
- the configuration:
** /etc/lava-server/ ** /etc/lava-dispatcher/certificates.d
- the job artifacts: /var/lib/lava-server/default/media/job-output
On the dispatchers, you should backup:
- dispatcher config: /etc/lava-dispatcher/
- all the changes you did like, /etc/ser2net.conf, udev rules, ...
Hope that helps.
Le jeu. 6 déc. 2018 à 18:35, Chris Sauer csauer@peoplenetonline.com a écrit :
Im deploying lava using the docker container and I am looking at the correct paths to mount as volumes as to not lose data between cluster rollovers. (and to have to backups) If I change the instance.conf to point to a database that is outside of the container that gets backed up and includes these paths in my volume mounts is that all I need to do? Or is there additional paths/files that should be included.
We haven't actually had a chance to test using an external database upstream yet. We will be looking at this and other similar issues in January and discussions will happen on the lava-devel mailing list (https://lists.lavasoftware.org/mailman/listinfo/lava-devel). More details will be announced soon.
For now, simply pointing the instance.conf elsewhere should work - it leaves a redundant but fully configured database in the container - but has not been tested. If you want to use a database outside the container, test your changes carefully on a staging or test instance and please let us know your results on the lava-devel mailing list.
https://master.lavasoftware.org/static/docs/v2/admin-backups.html
If anyone knows anything about this thanks!
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
-- Rémi Duraffort LAVA Team _______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
We moved forward with just exposing the data folder of the database as a volume because operations here already backs up all volumes automatically. But I will have two instances of lava so I can certainly still point the database of the dev instance to aws-RDS. I will post updates over in the lava-devel mailing list. I understand that the container deployment is a very new thing for lava but since we're just starting off with lava we won't have a lot to migrate if breaking changes occur. On Sun, Dec 9, 2018 at 11:47 PM Neil Williams neil.williams@linaro.org wrote:
On Fri, 7 Dec 2018 at 12:21, Remi Duraffort remi.duraffort@linaro.org wrote:
Hello,
when trying to backup a lava instance, keep in mind that lava is made of two parts : the server (only one) and the dispatchers (usually more than one).
For the server you should backup:
- the database (see /etc/lava-server/instance.conf)
- the configuration:
** /etc/lava-server/ ** /etc/lava-dispatcher/certificates.d
- the job artifacts: /var/lib/lava-server/default/media/job-output
On the dispatchers, you should backup:
- dispatcher config: /etc/lava-dispatcher/
- all the changes you did like, /etc/ser2net.conf, udev rules, ...
Hope that helps.
Le jeu. 6 déc. 2018 à 18:35, Chris Sauer csauer@peoplenetonline.com a écrit :
Im deploying lava using the docker container and I am looking at the correct paths to mount as volumes as to not lose data between cluster rollovers. (and to have to backups) If I change the instance.conf to point to a database that is outside of the container that gets backed up and includes these paths in my volume mounts is that all I need to do? Or is there additional paths/files that should be included.
We haven't actually had a chance to test using an external database upstream yet. We will be looking at this and other similar issues in January and discussions will happen on the lava-devel mailing list (https://lists.lavasoftware.org/mailman/listinfo/lava-devel). More details will be announced soon.
For now, simply pointing the instance.conf elsewhere should work - it leaves a redundant but fully configured database in the container - but has not been tested. If you want to use a database outside the container, test your changes carefully on a staging or test instance and please let us know your results on the lava-devel mailing list.
https://master.lavasoftware.org/static/docs/v2/admin-backups.html
If anyone knows anything about this thanks!
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
-- Rémi Duraffort LAVA Team _______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
--
Neil Williams
neil.williams@linaro.org http://www.linux.codehelp.co.uk/
On Mon, Dec 10, 2018 at 03:54:53PM -0600, Chris Sauer wrote:
We moved forward with just exposing the data folder of the database as a volume because operations here already backs up all volumes automatically. But I will have two instances of lava so I can certainly still point the database of the dev instance to aws-RDS. I will post updates over in the lava-devel mailing list. I understand that the container deployment is a very new thing for lava but since we're just starting off with lava we won't have a lot to migrate if breaking changes occur.
Hi Chris -
How did you do that? Are you rebuilding the container and modifying the entrypoint? I ask because it looks like the DB is already init'd in the image, and so if you mount in a volume over the top it will not re-init. I'm also wondering how you dealt with the DB path - hard coding '/var/lib/postgresql/9.6' seems fragile.
Thanks, Dan
The long term plan for me is to point the database using that file. What we ended up doing was copying the data outside of a container at `/var/lib/postgresql/9.6` to a AWS EBS volume and then we had todo a bunch of trickery to make sure those files stayed owned as psql user and ensure that the container starts in the same availability region. We are using Kubernetes to manage this so when we specify those EBS volumes in Kubernetes it mounts them before the container starts up, But we had to populate all the volumes with the defaults to start to make this work. Then we are able to make changes and EBS catches those changes. We are not planning on making any changes to the image at this time and only use whats provided until I can get a little more familiarized with Lava development. After I can get our device-types hooked in I do have a longer-term plan to hopefully add support for Kubernetes deployments out of the box with a little more separation in the containers, Allowing you to bring up a pod of all the services allowing you to omit stuff like the database if you want to use something like AWS-RDS instead of a psql-container but I'm not quite there yet.
On Wed, Dec 19, 2018 at 2:00 PM Dan Rue dan.rue@linaro.org wrote:
On Mon, Dec 10, 2018 at 03:54:53PM -0600, Chris Sauer wrote:
We moved forward with just exposing the data folder of the database as a volume because operations here already backs up all volumes automatically. But I will have two instances of lava so I can certainly still point the database of the dev instance to aws-RDS. I will post updates over in the lava-devel mailing list. I understand that the container deployment is a very new thing for lava but since we're just starting off with lava we won't have a lot to migrate if breaking changes occur.
Hi Chris -
How did you do that? Are you rebuilding the container and modifying the entrypoint? I ask because it looks like the DB is already init'd in the image, and so if you mount in a volume over the top it will not re-init. I'm also wondering how you dealt with the DB path - hard coding '/var/lib/postgresql/9.6' seems fragile.
Thanks, Dan
On Wed, Dec 19, 2018 at 02:36:51PM -0600, Chris Sauer wrote:
The long term plan for me is to point the database using that file. What we ended up doing was copying the data outside of a container at `/var/lib/postgresql/9.6` to a AWS EBS volume and then we had todo a bunch of trickery to make sure those files stayed owned as psql user and ensure that the container starts in the same availability region. We are using Kubernetes to manage this so when we specify those EBS volumes in Kubernetes it mounts them before the container starts up, But we had to populate all the volumes with the defaults to start to make this work. Then we are able to make changes and EBS catches those changes. We are not planning on making any changes to the image at this time and only use whats provided until I can get a little more familiarized with Lava development. After I can get our device-types hooked in I do have a longer-term plan to hopefully add support for Kubernetes deployments out of the box with a little more separation in the containers, Allowing you to bring up a pod of all the services allowing you to omit stuff like the database if you want to use something like AWS-RDS instead of a psql-container but I'm not quite there yet.
A few thoughts..
I just tried again using a separate container for the database and it seems to work. For some reason I didn't think it was supported. Have you tried? Are there any known issues using an external DB?
This solves another problem that I had with the official server container: Sometimes (often) on subsequent starts, the DB migration startup step would happen before postgres had fully started, causing startup to fail.
In the lava-server container though, if the initdb were moved into entrypoint.sh, then an empty directory can be mounted in the right location for persistence. As a part of this, perhaps use a location that doesn't contain a DB version number so that it will be more stable over time. Notably, the official postgres images allow the PGDATA env var to be set for this, and uses /var/lib/postgresql/data by default.
Dan
On Wed, Dec 19, 2018 at 2:00 PM Dan Rue dan.rue@linaro.org wrote:
On Mon, Dec 10, 2018 at 03:54:53PM -0600, Chris Sauer wrote:
We moved forward with just exposing the data folder of the database as a volume because operations here already backs up all volumes automatically. But I will have two instances of lava so I can certainly still point the database of the dev instance to aws-RDS. I will post updates over in the lava-devel mailing list. I understand that the container deployment is a very new thing for lava but since we're just starting off with lava we won't have a lot to migrate if breaking changes occur.
Hi Chris -
How did you do that? Are you rebuilding the container and modifying the entrypoint? I ask because it looks like the DB is already init'd in the image, and so if you mount in a volume over the top it will not re-init. I'm also wondering how you dealt with the DB path - hard coding '/var/lib/postgresql/9.6' seems fragile.
Thanks, Dan
Hello,
I just tried again using a separate container for the database and it
seems to work. For some reason I didn't think it was supported. Have you tried? Are there any known issues using an external DB?
I haven't tried yet but this should be working. You need to setup your external database with the right user and database. Then update /etc/lava-server/instance.conf to point to the right database. The lava-server container will still run postgresql process that you won't use.
This solves another problem that I had with the official server
container: Sometimes (often) on subsequent starts, the DB migration startup step would happen before postgres had fully started, causing startup to fail.
That's a bug then: https://git.lavasoftware.org/lava/pkg/docker/issues/6
Rgds
Hello,
This solves another problem that I had with the official server
container: Sometimes (often) on subsequent starts, the DB migration startup step would happen before postgres had fully started, causing startup to fail.
That's a bug then: https://git.lavasoftware.org/lava/pkg/docker/issues/6
I just pushed a new commit on lava/pkg/docker to update lava-server entrypoint. Now the script is waiting for postgresql to be up and running before doing the migrations.
When you have time, could you test this new entrypoint to see if that's fixing the issue.
Rgds
On Thu, Dec 20, 2018 at 05:02:13PM +0100, Remi Duraffort wrote:
Hello,
This solves another problem that I had with the official server
container: Sometimes (often) on subsequent starts, the DB migration startup step would happen before postgres had fully started, causing startup to fail.
That's a bug then: https://git.lavasoftware.org/lava/pkg/docker/issues/6
I just pushed a new commit on lava/pkg/docker to update lava-server entrypoint. Now the script is waiting for postgresql to be up and running before doing the migrations.
When you have time, could you test this new entrypoint to see if that's fixing the issue.
Tested and fixed @ https://git.lavasoftware.org/lava/pkg/docker/merge_requests/9. I do wonder if it should have a maximum wait time.
Thanks, Dan
Rgds
-- Rémi Duraffort LAVA Team
This solves another problem that I had with the official server
container: Sometimes (often) on subsequent starts, the DB migration startup step would happen before postgres had fully started, causing startup to fail.
That's a bug then:
https://git.lavasoftware.org/lava/pkg/docker/issues/6
I just pushed a new commit on lava/pkg/docker to update lava-server entrypoint. Now the script is waiting for postgresql to be up and running before doing the migrations.
When you have time, could you test this new entrypoint to see if that's fixing the issue.
Tested and fixed @ https://git.lavasoftware.org/lava/pkg/docker/merge_requests/9. I do wonder if it should have a maximum wait time.
Thanks Dan for the fix (merged)
Hello Dan,
How did you do that? Are you rebuilding the container and modifying
the entrypoint? I ask because it looks like the DB is already init'd in the image, and so if you mount in a volume over the top it will not re-init.
If you look at the lava-server entrypoint you will see that we call "lava-server manage migrate" ( https://git.lavasoftware.org/lava/pkg/docker/blob/master/amd64/lava-server/e...). If the user is able to connect to the database, this will force django to upgrade the schema or create it if it does not exist.
lava-users@lists.lavasoftware.org
-
Chris Sauer -
Dan Rue -
Neil Williams -
Remi Duraffort