Hi,
I've been using the lava-server docker image (hub.lavasoftware.org/lava/lava/lava-server:2018.10) for a couple of weeks and I had some problems on making the data persistent for the postgres (mapping the volume from host to container). Anyways I decided to take postgres out from the startup by modifying the entrypoint.sh file:
++++++++++++++ Added these lines after start_lava_server_gunicorn -function ++++++++++++++++++++++++++ if [[ ! -z $DJANGO_POSTGRES_SERVER ]]; then txt="s/LAVA_DB_SERVER="localhost"/LAVA_DB_SERVER="$DJANGO_POSTGRES_SERVER"/g" sed -i $txt /etc/lava-server/instance.conf fi
if [[ ! -z $DJANGO_POSTGRES_PORT ]]; then txt="s/LAVA_DB_PORT="5432"/LAVA_DB_PORT="$DJANGO_POSTGRES_PORT"/g" sed -i $txt /etc/lava-server/instance.conf fi ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
---------------------------- Commented out these lines ----------------------------- # Start all services #echo "Starting postgresql" #/etc/init.d/postgresql start #echo "done" #echo
#echo "Waiting for postgresql" #wait_postgresql #echo "[done]" #echo -------------------------------------------- ---------------------------------------------------- After that I created a new Dockerfile and built a new image:
+++++++++++++++++++++++++ Dockerfile +++++++++++++++++++++++ FROM hub.lavasoftware.org/lava/lava/lava-server:2018.10
COPY ./entrypoint.sh /root/ RUN chmod 755 /root/entrypoint.sh ENTRYPOINT ["/root/entrypoint.sh"] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
sudo docker build -t lava-server:mod .
Just to get all up and running I made a docker-compose.yml file to kick postgres and lava-server up
+++++++++++++++++++++++++++ docker-compose.yml +++++++++++++++++++++++++ version: '3' services: postgres: image: postgres:11 restart: always environment: POSTGRES_DB: lavaserver POSTGRES_USER: lavaserver POSTGRES_PASSWORD: d3e5d13fa15f lava-server: depends_on: - postgres image: lava-server:mod environment: DJANGO_POSTGRES_PORT: 5432 DJANGO_POSTGRES_SERVER: postgres ports: - "80:80" +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I still feel that there is too much going on inside that lava-server:mod image, since it has following softwares running:
* Lavamaster * Gunicorn * Logger * Publisher
What do you think, should I still break it into smaller pieces? Pros on this would be that softwares wouldn't die silently (started by using '&' and docker could try to restart them), logs would be in their own logs windows' (docker log CONTAINER) and containers themselves would be more configurable via environment variables.
Br Olli Väinölä IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
On Wed, 16 Jan 2019 at 16:56, Olli Vainola Olli.Vainola@arm.com wrote:
Hi,
I’ve been using the lava-server docker image (hub.lavasoftware.org/lava/lava/lava-server:2018.10) for a couple of weeks and I had some problems on making the data persistent for the postgres (mapping the volume from host to container). Anyways I decided to take postgres out from the startup by modifying the entrypoint.sh file:
++++++++++++++ Added these lines after start_lava_server_gunicorn -function ++++++++++++++++++++++++++
if [[ ! -z $DJANGO_POSTGRES_SERVER ]]; then
txt="s/LAVA_DB_SERVER=\"localhost\"/LAVA_DB_SERVER=\"$DJANGO_POSTGRES_SERVER\"/g" sed -i $txt /etc/lava-server/instance.conffi
if [[ ! -z $DJANGO_POSTGRES_PORT ]]; then
txt="s/LAVA_DB_PORT=\"5432\"/LAVA_DB_PORT=\"$DJANGO_POSTGRES_PORT\"/g" sed -i $txt /etc/lava-server/instance.conffi
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
---------------------------- Commented out these lines -----------------------------
# Start all services
#echo "Starting postgresql"
#/etc/init.d/postgresql start
#echo "done"
#echo
#echo "Waiting for postgresql"
#wait_postgresql
#echo "[done]"
#echo
After that I created a new Dockerfile and built a new image:
+++++++++++++++++++++++++ Dockerfile +++++++++++++++++++++++
FROM hub.lavasoftware.org/lava/lava/lava-server:2018.10
COPY ./entrypoint.sh /root/
RUN chmod 755 /root/entrypoint.sh
ENTRYPOINT ["/root/entrypoint.sh"]
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
sudo docker build -t lava-server:mod .
Just to get all up and running I made a docker-compose.yml file to kick postgres and lava-server up
+++++++++++++++++++++++++++ docker-compose.yml +++++++++++++++++++++++++
version: '3'
services:
postgres:
image: postgres:11 restart: always environment: POSTGRES_DB: lavaserver POSTGRES_USER: lavaserver POSTGRES_PASSWORD: d3e5d13fa15flava-server:
depends_on: - postgres image: lava-server:mod environment: DJANGO_POSTGRES_PORT: 5432 DJANGO_POSTGRES_SERVER: postgres ports: - "80:80"+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I still feel that there is too much going on inside that lava-server:mod image, since it has following softwares running:
Lavamaster Gunicorn Logger Publisher
Olli, Dan already did part of what you propose: https://github.com/danrue/lava-docker-compose I guess we should move the work to some common place not to repeat all the same steps
milosz
What do you think, should I still break it into smaller pieces? Pros on this would be that softwares wouldn’t die silently (started by using ‘&’ and docker could try to restart them), logs would be in their own logs windows’ (docker log CONTAINER) and containers themselves would be more configurable via environment variables.
Br
Olli Väinölä
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
On Wed, Jan 16, 2019 at 07:27:28PM +0000, Milosz Wasilewski wrote:
On Wed, 16 Jan 2019 at 16:56, Olli Vainola Olli.Vainola@arm.com wrote:
Hi,
I’ve been using the lava-server docker image (hub.lavasoftware.org/lava/lava/lava-server:2018.10) for a couple of weeks and I had some problems on making the data persistent for the postgres (mapping the volume from host to container). Anyways I decided to take postgres out from the startup by modifying the entrypoint.sh file:
++++++++++++++ Added these lines after start_lava_server_gunicorn -function ++++++++++++++++++++++++++
if [[ ! -z $DJANGO_POSTGRES_SERVER ]]; then
txt="s/LAVA_DB_SERVER=\"localhost\"/LAVA_DB_SERVER=\"$DJANGO_POSTGRES_SERVER\"/g" sed -i $txt /etc/lava-server/instance.conffi
if [[ ! -z $DJANGO_POSTGRES_PORT ]]; then
txt="s/LAVA_DB_PORT=\"5432\"/LAVA_DB_PORT=\"$DJANGO_POSTGRES_PORT\"/g" sed -i $txt /etc/lava-server/instance.conffi
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
---------------------------- Commented out these lines -----------------------------
# Start all services
#echo "Starting postgresql"
#/etc/init.d/postgresql start
#echo "done"
#echo
#echo "Waiting for postgresql"
#wait_postgresql
#echo "[done]"
#echo
After that I created a new Dockerfile and built a new image:
+++++++++++++++++++++++++ Dockerfile +++++++++++++++++++++++
FROM hub.lavasoftware.org/lava/lava/lava-server:2018.10
COPY ./entrypoint.sh /root/
RUN chmod 755 /root/entrypoint.sh
ENTRYPOINT ["/root/entrypoint.sh"]
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
sudo docker build -t lava-server:mod .
Just to get all up and running I made a docker-compose.yml file to kick postgres and lava-server up
+++++++++++++++++++++++++++ docker-compose.yml +++++++++++++++++++++++++
version: '3'
services:
postgres:
image: postgres:11 restart: always environment: POSTGRES_DB: lavaserver POSTGRES_USER: lavaserver POSTGRES_PASSWORD: d3e5d13fa15flava-server:
depends_on: - postgres image: lava-server:mod environment: DJANGO_POSTGRES_PORT: 5432 DJANGO_POSTGRES_SERVER: postgres ports: - "80:80"+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I still feel that there is too much going on inside that lava-server:mod image, since it has following softwares running:
Lavamaster Gunicorn Logger Publisher
Olli, Dan already did part of what you propose: https://github.com/danrue/lava-docker-compose I guess we should move the work to some common place not to repeat all the same steps
Yes, it sounds like Olli and I are on the same path :)
I didn't bother modifying the container though. I just mounted in my own copy of instance.conf. Adding environment variable support, as you have proposed, for common config properties would be better. This meant that postgresql is still running an empty database in my server container for the time being.
By the way, I noticed you're using postgres 11, yet the lava-server container uses 9.6. It's probably fine? I have no idea - I tried to keep the version consistent with what's tested and supported. Also, in case you need it, there was a fix to the entrypoint to properly wait for postgres. Find the latest entrypoint at https://git.lavasoftware.org/lava/pkg/docker/blob/master/amd64/lava-server/e...
What do you think, should I still break it into smaller pieces? Pros on this would be that softwares wouldn’t die silently (started by using ‘&’ and docker could try to restart them), logs would be in their own logs windows’ (docker log CONTAINER) and containers themselves would be more configurable via environment variables.
I agree in principle, but I'm not sure of the implications for each of the services. For the time being, I'm focusing on trying to use the containers as they are, and upstream improvements to make them easier to use.
I do think I'll try using a separate ser2net container with my dispatcher though - it seems like an easy case, ser2net's already not started by default, and I want a newer version of ser2net than what's installed in the dispatcher by default.
Olli - in light of all that, I'm curious about your feedback on this merge request: https://git.lavasoftware.org/lava/pkg/docker/merge_requests/10.
Thanks! Dan
Br
Olli Väinölä
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
Hi,
-----Original Message----- From: Dan Rue dan.rue@linaro.org Sent: Wednesday, January 16, 2019 10:46 PM To: Milosz Wasilewski milosz.wasilewski@linaro.org Cc: Olli Vainola Olli.Vainola@arm.com; lava-users@lists.lavasoftware.org Subject: Re: [Lava-users] Splitting the lava-server docker image
On Wed, Jan 16, 2019 at 07:27:28PM +0000, Milosz Wasilewski wrote:
On Wed, 16 Jan 2019 at 16:56, Olli Vainola Olli.Vainola@arm.com wrote:
Hi,
I’ve been using the lava-server docker image (hub.lavasoftware.org/lava/lava/lava-server:2018.10) for a couple of weeks and I had some problems on making the data persistent for the postgres (mapping the volume from host to container). Anyways I decided to take postgres out from the startup by modifying the entrypoint.sh file:
++++++++++++++ Added these lines after start_lava_server_gunicorn ++++++++++++++ -function ++++++++++++++++++++++++++
if [[ ! -z $DJANGO_POSTGRES_SERVER ]]; then
txt="s/LAVA_DB_SERVER=\"localhost\"/LAVA_DB_SERVER=\"$DJANGO_POSTGRES_SERVER\"/g" sed -i $txt /etc/lava-server/instance.conffi
if [[ ! -z $DJANGO_POSTGRES_PORT ]]; then
txt="s/LAVA_DB_PORT=\"5432\"/LAVA_DB_PORT=\"$DJANGO_POSTGRES_PORT\"/g" sed -i $txt /etc/lava-server/instance.conffi
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
---------------------------- Commented out these lines
# Start all services
#echo "Starting postgresql"
#/etc/init.d/postgresql start
#echo "done"
#echo
#echo "Waiting for postgresql"
#wait_postgresql
#echo "[done]"
#echo
After that I created a new Dockerfile and built a new image:
+++++++++++++++++++++++++ Dockerfile +++++++++++++++++++++++
FROM hub.lavasoftware.org/lava/lava/lava-server:2018.10
COPY ./entrypoint.sh /root/
RUN chmod 755 /root/entrypoint.sh
ENTRYPOINT ["/root/entrypoint.sh"]
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
sudo docker build -t lava-server:mod .
Just to get all up and running I made a docker-compose.yml file to kick postgres and lava-server up
+++++++++++++++++++++++++++ docker-compose.yml +++++++++++++++++++++++++++ +++++++++++++++++++++++++
version: '3'
services:
postgres:
image: postgres:11 restart: always environment: POSTGRES_DB: lavaserver POSTGRES_USER: lavaserver POSTGRES_PASSWORD: d3e5d13fa15flava-server:
depends_on: - postgres image: lava-server:mod environment: DJANGO_POSTGRES_PORT: 5432 DJANGO_POSTGRES_SERVER: postgres ports: - "80:80"+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I still feel that there is too much going on inside that lava-server:mod image, since it has following softwares running:
Lavamaster Gunicorn Logger Publisher
Olli, Dan already did part of what you propose: https://github.com/danrue/lava-docker-compose I guess we should move the work to some common place not to repeat all the same steps
Yes, it sounds like Olli and I are on the same path :)
I didn't bother modifying the container though. I just mounted in my own copy of instance.conf. Adding environment variable support, as you have proposed, for common config properties would be better. This meant that postgresql is still running an empty database in my server container for the time being.
I was also thinking on injecting the configuration file via volume mount, but then again postgres was running there for no reason, so I figured just to modify the entrypoint.sh itself :). A bit hacky hacky but did the trick.
By the way, I noticed you're using postgres 11, yet the lava-server container uses 9.6. It's probably fine? I have no idea - I tried to keep the version consistent with what's tested and supported. Also, in case you need it, there was a fix to the entrypoint to properly wait for postgres. Find the latest entrypoint at https://git.lavasoftware.org/lava/pkg/docker/blob/master/amd64/lava-server/e...
At least the data migration part went ok, so I'll go with a strong maybe. Web interface responded properly but I didn't add slaves yet, so haven't tested it fully. I also found that entrypoint.sh from the gitlab and yes, there was some timing issues when postgres didn't start fast enough and lava-server crashed. Rather than wait, there could be some sanity check for connection with some retry & timeout functionality.
What do you think, should I still break it into smaller pieces? Pros on this would be that softwares wouldn’t die silently (started by using ‘&’ and docker could try to restart them), logs would be in their own logs windows’ (docker log CONTAINER) and containers themselves would be more configurable via environment variables.
I agree in principle, but I'm not sure of the implications for each of the services. For the time being, I'm focusing on trying to use the containers as they are, and upstream improvements to make them easier to use.
I can try to separate all server side services and see if any major issues arise.
I do think I'll try using a separate ser2net container with my dispatcher though - it seems like an easy case, ser2net's already not started by default, and I want a newer version of ser2net than what's installed in the dispatcher by default.
Olli - in light of all that, I'm curious about your feedback on this merge request: https://git.lavasoftware.org/lava/pkg/docker/merge_requests/10.
Looks great! That would make the starting the container more generic and one could add more functionality without building the container.
Br Olli
Thanks! Dan
Br
Olli Väinölä
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
-- Linaro - Kernel Validation
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Hello,
Dan already did part of what you propose:
https://github.com/danrue/lava-docker-compose I guess we should move the work to some common place not to repeat all the same steps
Yes, it sounds like Olli and I are on the same path :)
I didn't bother modifying the container though. I just mounted in my own
copy of instance.conf. Adding environment variable support, as you have proposed, for common config properties would be better. This meant that postgresql is still running an empty database in my server container for the time being.
Dan pushed an RFC allowing to inject custom entrypoints that would be called after the startup of every services. See https://git.lavasoftware.org/lava/pkg/docker/merge_requests/10
By the way, I noticed you're using postgres 11, yet the lava-server
container uses 9.6. It's probably fine? I have no idea - I tried to keep the version consistent with what's tested and supported. Also, in case you need it, there was a fix to the entrypoint to properly wait for postgres. Find the latest entrypoint at https://git.lavasoftware.org/lava/pkg/docker/blob/master/amd64/lava-server/e...
At least the data migration part went ok, so I'll go with a strong maybe. Web interface responded properly but I didn't add slaves yet, so haven't tested it fully. I also found that entrypoint.sh from the gitlab and yes, there was some timing issues when postgres didn't start fast enough and lava-server crashed. Rather than wait, there could be some sanity check for connection with some retry & timeout functionality.
The last version of the entrypoint is waiting properly for postgresql startup by trying to connect to pgsql (and retrying until that success). See https://git.lavasoftware.org/lava/pkg/docker/blob/master/amd64/lava-server/e...
What do you think, should I still break it into smaller pieces? Pros
on this would be that softwares wouldn’t die silently (started by using ‘&’ and docker could try to restart them), logs would be in their own logs windows’ (docker log CONTAINER) and containers themselves would be more configurable via environment variables.
I agree in principle, but I'm not sure of the implications for each of
the services. For the time being, I'm focusing on trying to use the containers as they are, and upstream improvements to make them easier to use.
I can try to separate all server side services and see if any major issues arise.
We are thinking about providing a way to only start some services (via env variable I guess). So you can use the official docker images and start only some services.
The docker files are in: https://git.lavasoftware.org/lava/pkg/docker/ Don't hesitate to send merge requests if needed or to fill bug reports.
Cheers
lava-users@lists.lavasoftware.org
-
Dan Rue -
Milosz Wasilewski -
Olli Vainola -
Remi Duraffort