Hello everyone,
I added an email notification to a test job but forgot to configure an SMTP server first. The job reported:
"JobError: Your job cannot terminate cleanly."
Afterwards nothing happened. The job was still running, even after all timeouts had been passed, so I tried to cancel it. Now the job remains in "Cancelling" state and I have no idea why and how to fix this. Any hints?
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks@garz-fricke.com www.garz-fricke.com SOLUTIONS THAT COMPLETE!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz
On 24 April 2018 at 12:05, Tim Jaacks tim.jaacks@garz-fricke.com wrote:
Hello everyone,
I added an email notification to a test job but forgot to configure an SMTP server first.
There will likely be an exception logged in /var/log/lava-server/lava-master.log
You will need to restart the lava-master
$ sudo service lava-master restart
Then check the logs.
You don't need an SMTP server to be configured in LAVA but you do need a working mail configuration on the master so that you can send email to external addresses using the command line program, mail. The SMTP settings in lava-server are an alternative. If you use the external mail support, mail errors would remain in the email system of the master.
The job reported:
"JobError: Your job cannot terminate cleanly."
Afterwards nothing happened. The job was still running, even after all timeouts had been passed, so I tried to cancel it. Now the job remains in "Cancelling" state and I have no idea why and how to fix this. Any hints?
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks@garz-fricke.com www.garz-fricke.com SOLUTIONS THAT COMPLETE!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz
Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
Hi Neil,
thanks for your quick help.
Hello everyone,
I added an email notification to a test job but forgot to configure an SMTP server first.
There will likely be an exception logged in /var/log/lava-server/lava-master.log
You will need to restart the lava-master
$ sudo service lava-master restart
Then check the logs.
My log says:
2018-04-24 14:22:08,025 INFO [POLL] Received a signal, leaving 2018-04-24 14:22:08,025 INFO [CLOSE] Closing the controler socket and dropping messages 2018-04-24 14:22:09,183 INFO [INIT] Dropping privileges 2018-04-24 14:22:09,184 DEBUG Switching to (lavaserver(114), lavaserver(118)) 2018-04-24 14:22:09,184 INFO [INIT] Marking all workers as offline 2018-04-24 14:22:09,191 INFO [INIT] LAVA master has started. 2018-04-24 14:22:09,191 INFO [INIT] Using protocol version 3 2018-04-24 14:22:23,300 DEBUG lava-logs => PING(20) 2018-04-24 14:22:25,485 DEBUG A048.HAMBURG => PING(20) 2018-04-24 14:22:25,485 WARNING Unknown dispatcher <A048.HAMBURG> (server crashed) 2018-04-24 14:22:29,195 INFO scheduling health checks: 2018-04-24 14:22:29,234 DEBUG -> disabled on: lxc, qemu 2018-04-24 14:22:29,235 INFO scheduling jobs: 2018-04-24 14:22:29,236 DEBUG - beaglebone-black 2018-04-24 14:22:29,246 DEBUG - imx6guf 2018-04-24 14:22:29,255 DEBUG - lxc 2018-04-24 14:22:29,263 DEBUG - qemu 2018-04-24 14:22:29,284 INFO [357] CANCEL => A048.HAMBURG 2018-04-24 14:22:29,287 INFO [357] A048.HAMBURG => END (duplicated), skipping
What does this mean? FYI: Master and dispatcher are on the same machine. Can I restart the dispatcher somehow?
You don't need an SMTP server to be configured in LAVA but you do need a working mail configuration on the master so that you can send email to external addresses using the command line program, mail. The SMTP settings in lava-server are an alternative. If you use the external mail support, mail errors would remain in the email system of the master.
The job reported:
"JobError: Your job cannot terminate cleanly."
Afterwards nothing happened. The job was still running, even after all timeouts had been passed, so I tried to cancel it. Now the job remains in "Cancelling" state and I have no idea why and how to fix this. Any hints?
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks at garz-fricke.com www.garz-fricke.com SOLUTIONS THAT COMPLETE!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz
Lava-users mailing list Lava-users at lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
--
Neil Williams
neil.williams at linaro.org http://www.linux.codehelp.co.uk/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.linaro.org/pipermail/lava-users/attachments/20180424/e7a046b0/attachment.html
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks@garz-fricke.com www.garz-fricke.com SOLUTIONS THAT COMPLETE!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz
Hey everyone,
I still have this problem. Can anyone assist me? Rebooting the server does not help. :-/
Regards, Tim
Hi Neil,
thanks for your quick help.
Hello everyone,
I added an email notification to a test job but forgot to configure an SMTP server first.
There will likely be an exception logged in /var/log/lava-server/lava-master.log
You will need to restart the lava-master
$ sudo service lava-master restart
Then check the logs.
My log says:
2018-04-24 14:22:08,025 INFO [POLL] Received a signal, leaving 2018-04-24 14:22:08,025 INFO [CLOSE] Closing the controler socket and dropping messages 2018-04-24 14:22:09,183 INFO [INIT] Dropping privileges 2018-04-24 14:22:09,184 DEBUG Switching to (lavaserver(114), lavaserver(118)) 2018-04-24 14:22:09,184 INFO [INIT] Marking all workers as offline 2018-04-24 14:22:09,191 INFO [INIT] LAVA master has started. 2018-04-24 14:22:09,191 INFO [INIT] Using protocol version 3 2018-04-24 14:22:23,300 DEBUG lava-logs => PING(20) 2018-04-24 14:22:25,485 DEBUG A048.HAMBURG => PING(20) 2018-04-24 14:22:25,485 WARNING Unknown dispatcher <A048.HAMBURG> (server crashed) 2018-04-24 14:22:29,195 INFO scheduling health checks: 2018-04-24 14:22:29,234 DEBUG -> disabled on: lxc, qemu 2018-04-24 14:22:29,235 INFO scheduling jobs: 2018-04-24 14:22:29,236 DEBUG - beaglebone-black 2018-04-24 14:22:29,246 DEBUG - imx6guf 2018-04-24 14:22:29,255 DEBUG - lxc 2018-04-24 14:22:29,263 DEBUG - qemu 2018-04-24 14:22:29,284 INFO [357] CANCEL => A048.HAMBURG 2018-04-24 14:22:29,287 INFO [357] A048.HAMBURG => END (duplicated), skipping
What does this mean? FYI: Master and dispatcher are on the same machine. Can I restart the dispatcher somehow?
You don't need an SMTP server to be configured in LAVA but you do need a working mail configuration on the master so that you can send email to external addresses using the command line program, mail. The SMTP settings in lava-server are an alternative. If you use the external mail support, mail errors would remain in the email system of the master.
The job reported:
"JobError: Your job cannot terminate cleanly."
Afterwards nothing happened. The job was still running, even after all timeouts had been passed, so I tried to cancel it. Now the job remains in "Cancelling" state and I have no idea why and how to fix this. Any hints?
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks at garz-fricke.com www.garz-fricke.com SOLUTIONS THAT COMPLETE!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz
Lava-users mailing list Lava-users at lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
--
Neil Williams
neil.williams at linaro.org http://www.linux.codehelp.co.uk/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.linaro.org/pipermail/lava-users/attachments/20180424/e7a0 46b0/attachment.html
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks@garz-fricke.com www.garz-fricke.com SOLUTIONS THAT COMPLETE!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz _______________________________________________ Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
Hello,
rebooting the server will not fix anything as the status is saved into the database (so kept between reboots).
1/ Are lava-master, lava-slave and lava-logs all running?
2/ Do you see anything in the corresponding logs? (/var/log/lava-server/lava-master.log, /var/log/lava-server/lava-logs.log, /var/log/lava-dispatcher/lava-slave.log)
3/ Which version of lava re you using?
Regards
2018-04-30 10:14 GMT+02:00 Tim Jaacks tim.jaacks@garz-fricke.com:
Hey everyone,
I still have this problem. Can anyone assist me? Rebooting the server does not help. :-/
Regards, Tim
Hi Neil,
thanks for your quick help.
Hello everyone,
I added an email notification to a test job but forgot to configure an SMTP server first.
There will likely be an exception logged in /var/log/lava-server/lava-master.log
You will need to restart the lava-master
$ sudo service lava-master restart
Then check the logs.
My log says:
2018-04-24 14:22:08,025 INFO [POLL] Received a signal, leaving 2018-04-24 14:22:08,025 INFO [CLOSE] Closing the controler socket and
dropping messages
2018-04-24 14:22:09,183 INFO [INIT] Dropping privileges 2018-04-24 14:22:09,184 DEBUG Switching to (lavaserver(114),
lavaserver(118))
2018-04-24 14:22:09,184 INFO [INIT] Marking all workers as offline 2018-04-24 14:22:09,191 INFO [INIT] LAVA master has started. 2018-04-24 14:22:09,191 INFO [INIT] Using protocol version 3 2018-04-24 14:22:23,300 DEBUG lava-logs => PING(20) 2018-04-24 14:22:25,485 DEBUG A048.HAMBURG => PING(20) 2018-04-24 14:22:25,485 WARNING Unknown dispatcher <A048.HAMBURG> (server
crashed)
2018-04-24 14:22:29,195 INFO scheduling health checks: 2018-04-24 14:22:29,234 DEBUG -> disabled on: lxc, qemu 2018-04-24 14:22:29,235 INFO scheduling jobs: 2018-04-24 14:22:29,236 DEBUG - beaglebone-black 2018-04-24 14:22:29,246 DEBUG - imx6guf 2018-04-24 14:22:29,255 DEBUG - lxc 2018-04-24 14:22:29,263 DEBUG - qemu 2018-04-24 14:22:29,284 INFO [357] CANCEL => A048.HAMBURG 2018-04-24 14:22:29,287 INFO [357] A048.HAMBURG => END (duplicated),
skipping
What does this mean? FYI: Master and dispatcher are on the same machine.
Can I restart the dispatcher somehow?
You don't need an SMTP server to be configured in LAVA but you do need a working mail configuration on the master so that you can send email to external addresses using the command line program, mail. The SMTP settings in lava-server are an alternative. If you use the external mail support, mail errors would remain in the email system of the master.
The job reported:
"JobError: Your job cannot terminate cleanly."
Afterwards nothing happened. The job was still running, even after all timeouts had been passed, so I tried to cancel it. Now the job remains in "Cancelling" state and I have no idea why and how to fix
this. Any hints?
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks at garz-fricke.com www.garz-fricke.com SOLUTIONS THAT COMPLETE!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz
Lava-users mailing list Lava-users at lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
--
Neil Williams
neil.williams at linaro.org http://www.linux.codehelp.co.uk/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.linaro.org/pipermail/lava-users/attachments/20180424/e7a0 46b0/attachment.html
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks@garz-fricke.com www.garz-fricke.com SOLUTIONS THAT COMPLETE!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz
Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
Hi Remi,
thanks for your quick reply.
Hello,
rebooting the server will not fix anything as the status is saved into the database (so kept between reboots).
1/ Are lava-master, lava-slave and lava-logs all running?
I think so, yes:
tim.jaacks@A048:~$ ps -ax | grep lava 637 ? Ssl 0:01 /usr/bin/python /usr/bin/lava-server manage lava-publisher --level DEBUG 673 ? Ssl 0:41 /usr/bin/python /usr/bin/lava-server manage lava-logs --level DEBUG 678 ? Ssl 1:26 /usr/bin/python /usr/bin/lava-slave --level DEBUG --master tcp://localhost:5556 --socket-addr tcp://localhost:5555 824 ? S 0:00 /usr/bin/python /usr/bin/lava-coordinator --loglevel=DEBUG 1017 ? S 9:30 /usr/bin/python /usr/sbin/lavapdu-runner --loglevel=INFO 1018 ? S 0:51 /usr/bin/python /usr/sbin/lavapdu-listen --loglevel=INFO 1473 ? Ss 0:00 postgres: 9.6/main: lavaserver lavaserver ::1(35670) idle 2476 ? Ss 1:19 gunicorn: master [lava_server.wsgi] 2482 ? S 0:12 gunicorn: worker [lava_server.wsgi] 2483 ? S 0:11 gunicorn: worker [lava_server.wsgi] 2484 ? S 0:13 gunicorn: worker [lava_server.wsgi] 2489 ? S 0:13 gunicorn: worker [lava_server.wsgi] 3077 ? Ssl 24:59 /usr/bin/python /usr/bin/lava-server manage lava-master --level DEBUG 3079 ? Ss 8:13 postgres: 9.6/main: lavaserver lavaserver ::1(38208) idle 3418 pts/0 S+ 0:00 grep lava
2/ Do you see anything in the corresponding logs? (/var/log/lava-server/lava-master.log, /var/log/lava-server/lava-logs.log, /var/log/lava-dispatcher/lava-slave.log)
Oh yes, there is something in lava-logs.log:
2018-04-30 11:00:52,046 DEBUG PING => master (last message 20s ago) 2018-04-30 11:00:52,106 INFO master => CANCEL(357) 2018-04-30 11:00:52,106 DEBUG [357] Unknown job, sending END 2018-04-30 11:00:52,107 ERROR [357] Unable to read 'description.yaml' 2018-04-30 11:00:52,107 ERROR [Errno 2] No such file or directory: '/var/lib/lava/dispatcher/slave/tmp/357/description.yaml' Traceback (most recent call last): File "/usr/bin/lava-slave", line 180, in description data = open(filename, 'r').read() IOError: [Errno 2] No such file or directory: '/var/lib/lava/dispatcher/slave/tmp/357/description.yaml' 2018-04-30 11:00:52,107 ERROR [357] lava-run crashed 2018-04-30 11:00:52,107 DEBUG master => PONG(20) 2018-04-30 11:00:52,114 INFO master => END_OK(357) 2018-04-30 11:00:52,114 DEBUG [357] Job END acked 2018-04-30 11:00:52,114 DEBUG [357] Removing /var/lib/lava/dispatcher/slave/tmp/357
Can you tell what this means?
3/ Which version of lava re you using?
2018.2
Regards
2018-04-30 10:14 GMT+02:00 Tim Jaacks tim.jaacks@garz-fricke.com: Hey everyone,
I still have this problem. Can anyone assist me? Rebooting the server does not help. :-/
Regards, Tim
Hi Neil,
thanks for your quick help.
Hello everyone,
I added an email notification to a test job but forgot to configure an SMTP server first.
There will likely be an exception logged in /var/log/lava-server/lava-master.log
You will need to restart the lava-master
$ sudo service lava-master restart
Then check the logs.
My log says:
2018-04-24 14:22:08,025 INFO [POLL] Received a signal, leaving 2018-04-24 14:22:08,025 INFO [CLOSE] Closing the controler socket and dropping messages 2018-04-24 14:22:09,183 INFO [INIT] Dropping privileges 2018-04-24 14:22:09,184 DEBUG Switching to (lavaserver(114), lavaserver(118)) 2018-04-24 14:22:09,184 INFO [INIT] Marking all workers as offline 2018-04-24 14:22:09,191 INFO [INIT] LAVA master has started. 2018-04-24 14:22:09,191 INFO [INIT] Using protocol version 3 2018-04-24 14:22:23,300 DEBUG lava-logs => PING(20) 2018-04-24 14:22:25,485 DEBUG A048.HAMBURG => PING(20) 2018-04-24 14:22:25,485 WARNING Unknown dispatcher <A048.HAMBURG> (server crashed) 2018-04-24 14:22:29,195 INFO scheduling health checks: 2018-04-24 14:22:29,234 DEBUG -> disabled on: lxc, qemu 2018-04-24 14:22:29,235 INFO scheduling jobs: 2018-04-24 14:22:29,236 DEBUG - beaglebone-black 2018-04-24 14:22:29,246 DEBUG - imx6guf 2018-04-24 14:22:29,255 DEBUG - lxc 2018-04-24 14:22:29,263 DEBUG - qemu 2018-04-24 14:22:29,284 INFO [357] CANCEL => A048.HAMBURG 2018-04-24 14:22:29,287 INFO [357] A048.HAMBURG => END (duplicated), skipping
What does this mean? FYI: Master and dispatcher are on the same machine. Can I restart the dispatcher somehow?
You don't need an SMTP server to be configured in LAVA but you do need a working mail configuration on the master so that you can send email to external addresses using the command line program, mail. The SMTP settings in lava-server are an alternative. If you use the external mail support, mail errors would remain in the email system of the master.
The job reported:
"JobError: Your job cannot terminate cleanly."
Afterwards nothing happened. The job was still running, even after all timeouts had been passed, so I tried to cancel it. Now the job remains in "Cancelling" state and I have no idea why and how to fix this. Any hints?
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks at garz-fricke.com www.garz-fricke.com SOLUTIONS THAT COMPLETE!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz
Lava-users mailing list Lava-users at lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
--
Neil Williams
neil.williams at linaro.org http://www.linux.codehelp.co.uk/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.linaro.org/pipermail/lava-users/attachments/20180424/e7a0 46b0/attachment.html
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks@garz-fricke.com www.garz-fricke.com SOLUTIONS THAT COMPLETE!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz _______________________________________________ Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
-- Rémi Duraffort LAVA Team
If you are sure that the jobs is finished (the corresponding lava-run is not running), you can force a job to finish by using "lava-server manage jobs fail <job_id>".
This will change the job and device state in the database directly. This is useful if something went wrong and you are sure that lava-run has terminated.
Regards
2018-04-30 13:04 GMT+02:00 Tim Jaacks tim.jaacks@garz-fricke.com:
Hi Remi,
thanks for your quick reply.
Hello,
rebooting the server will not fix anything as the status is saved into
the database (so kept between reboots).
1/ Are lava-master, lava-slave and lava-logs all running?
I think so, yes:
tim.jaacks@A048:~$ ps -ax | grep lava 637 ? Ssl 0:01 /usr/bin/python /usr/bin/lava-server manage lava-publisher --level DEBUG 673 ? Ssl 0:41 /usr/bin/python /usr/bin/lava-server manage lava-logs --level DEBUG 678 ? Ssl 1:26 /usr/bin/python /usr/bin/lava-slave --level DEBUG --master tcp://localhost:5556 --socket-addr tcp://localhost:5555 824 ? S 0:00 /usr/bin/python /usr/bin/lava-coordinator --loglevel=DEBUG 1017 ? S 9:30 /usr/bin/python /usr/sbin/lavapdu-runner --loglevel=INFO 1018 ? S 0:51 /usr/bin/python /usr/sbin/lavapdu-listen --loglevel=INFO 1473 ? Ss 0:00 postgres: 9.6/main: lavaserver lavaserver ::1(35670) idle 2476 ? Ss 1:19 gunicorn: master [lava_server.wsgi] 2482 ? S 0:12 gunicorn: worker [lava_server.wsgi] 2483 ? S 0:11 gunicorn: worker [lava_server.wsgi] 2484 ? S 0:13 gunicorn: worker [lava_server.wsgi] 2489 ? S 0:13 gunicorn: worker [lava_server.wsgi] 3077 ? Ssl 24:59 /usr/bin/python /usr/bin/lava-server manage lava-master --level DEBUG 3079 ? Ss 8:13 postgres: 9.6/main: lavaserver lavaserver ::1(38208) idle 3418 pts/0 S+ 0:00 grep lava
2/ Do you see anything in the corresponding logs?
(/var/log/lava-server/lava-master.log, /var/log/lava-server/lava-logs.log, /var/log/lava-dispatcher/lava-slave.log)
Oh yes, there is something in lava-logs.log:
2018-04-30 11:00:52,046 DEBUG PING => master (last message 20s ago) 2018-04-30 11:00:52,106 INFO master => CANCEL(357) 2018-04-30 11:00:52,106 DEBUG [357] Unknown job, sending END 2018-04-30 11:00:52,107 ERROR [357] Unable to read 'description.yaml' 2018-04-30 11:00:52,107 ERROR [Errno 2] No such file or directory: '/var/lib/lava/dispatcher/slave/tmp/357/description.yaml' Traceback (most recent call last): File "/usr/bin/lava-slave", line 180, in description data = open(filename, 'r').read() IOError: [Errno 2] No such file or directory: '/var/lib/lava/dispatcher/ slave/tmp/357/description.yaml' 2018-04-30 11:00:52,107 ERROR [357] lava-run crashed 2018-04-30 11:00:52,107 DEBUG master => PONG(20) 2018-04-30 11:00:52,114 INFO master => END_OK(357) 2018-04-30 11:00:52,114 DEBUG [357] Job END acked 2018-04-30 11:00:52,114 DEBUG [357] Removing /var/lib/lava/dispatcher/ slave/tmp/357
Can you tell what this means?
3/ Which version of lava re you using?
2018.2
Regards
2018-04-30 10:14 GMT+02:00 Tim Jaacks tim.jaacks@garz-fricke.com: Hey everyone,
I still have this problem. Can anyone assist me? Rebooting the server
does not help. :-/
Regards, Tim
Hi Neil,
thanks for your quick help.
Hello everyone,
I added an email notification to a test job but forgot to configure an SMTP server first.
There will likely be an exception logged in /var/log/lava-server/lava-master.log
You will need to restart the lava-master
$ sudo service lava-master restart
Then check the logs.
My log says:
2018-04-24 14:22:08,025 INFO [POLL] Received a signal, leaving 2018-04-24 14:22:08,025 INFO [CLOSE] Closing the controler socket and
dropping messages
2018-04-24 14:22:09,183 INFO [INIT] Dropping privileges 2018-04-24 14:22:09,184 DEBUG Switching to (lavaserver(114),
lavaserver(118))
2018-04-24 14:22:09,184 INFO [INIT] Marking all workers as offline 2018-04-24 14:22:09,191 INFO [INIT] LAVA master has started. 2018-04-24 14:22:09,191 INFO [INIT] Using protocol version 3 2018-04-24 14:22:23,300 DEBUG lava-logs => PING(20) 2018-04-24 14:22:25,485 DEBUG A048.HAMBURG => PING(20) 2018-04-24 14:22:25,485 WARNING Unknown dispatcher <A048.HAMBURG>
(server crashed)
2018-04-24 14:22:29,195 INFO scheduling health checks: 2018-04-24 14:22:29,234 DEBUG -> disabled on: lxc, qemu 2018-04-24 14:22:29,235 INFO scheduling jobs: 2018-04-24 14:22:29,236 DEBUG - beaglebone-black 2018-04-24 14:22:29,246 DEBUG - imx6guf 2018-04-24 14:22:29,255 DEBUG - lxc 2018-04-24 14:22:29,263 DEBUG - qemu 2018-04-24 14:22:29,284 INFO [357] CANCEL => A048.HAMBURG 2018-04-24 14:22:29,287 INFO [357] A048.HAMBURG => END (duplicated),
skipping
What does this mean? FYI: Master and dispatcher are on the same machine.
Can I restart the dispatcher somehow?
You don't need an SMTP server to be configured in LAVA but you do need a working mail configuration on the master so that you can send email to external addresses using the command line program, mail. The SMTP settings in lava-server are an alternative. If you use the external mail support, mail errors would remain in the email system of the
master.
The job reported:
"JobError: Your job cannot terminate cleanly."
Afterwards nothing happened. The job was still running, even after all timeouts had been passed, so I tried to cancel it. Now the job remains in "Cancelling" state and I have no idea why and how to fix
this. Any hints?
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks at garz-fricke.com www.garz-fricke.com SOLUTIONS THAT COMPLETE!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz
Lava-users mailing list Lava-users at lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
--
Neil Williams
neil.williams at linaro.org http://www.linux.codehelp.co.uk/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.linaro.org/pipermail/lava-users/attachments/20180424/e7a0 46b0/attachment.html
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks@garz-fricke.com www.garz-fricke.com SOLUTIONS THAT COMPLETE!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz
Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
-- Rémi Duraffort LAVA Team
Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
If you are sure that the jobs is finished (the corresponding lava-run is not running), you can force a job to finish by using "lava-server manage jobs fail <job_id
".
This will change the job and device state in the database directly. This is useful if something went wrong and you are sure that lava-run has terminated.
Thanks Remi, that helped.
Regards
2018-04-30 13:04 GMT+02:00 Tim Jaacks <tim.jaacks@garz-fricke.com
:
Hi Remi,
thanks for your quick reply.
Hello,
rebooting the server will not fix anything as the status is saved into the database (so kept between reboots).
1/ Are lava-master, lava-slave and lava-logs all running?
I think so, yes:
tim.jaacks@A048:~$ ps -ax | grep lava 637 ? Ssl 0:01 /usr/bin/python /usr/bin/lava-server manage lava-publisher --level DEBUG 673 ? Ssl 0:41 /usr/bin/python /usr/bin/lava-server manage lava-logs --level DEBUG 678 ? Ssl 1:26 /usr/bin/python /usr/bin/lava-slave --level DEBUG --master tcp://localhost:5556 --socket-addr tcp://localhost:5555 824 ? S 0:00 /usr/bin/python /usr/bin/lava-coordinator --loglevel=DEBUG 1017 ? S 9:30 /usr/bin/python /usr/sbin/lavapdu-runner --loglevel=INFO 1018 ? S 0:51 /usr/bin/python /usr/sbin/lavapdu-listen --loglevel=INFO 1473 ? Ss 0:00 postgres: 9.6/main: lavaserver lavaserver ::1(35670) idle 2476 ? Ss 1:19 gunicorn: master [lava_server.wsgi] 2482 ? S 0:12 gunicorn: worker [lava_server.wsgi] 2483 ? S 0:11 gunicorn: worker [lava_server.wsgi] 2484 ? S 0:13 gunicorn: worker [lava_server.wsgi] 2489 ? S 0:13 gunicorn: worker [lava_server.wsgi] 3077 ? Ssl 24:59 /usr/bin/python /usr/bin/lava-server manage lava-master --level DEBUG 3079 ? Ss 8:13 postgres: 9.6/main: lavaserver lavaserver ::1(38208) idle 3418 pts/0 S+ 0:00 grep lava
2/ Do you see anything in the corresponding logs? (/var/log/lava-server/lava-master.log, /var/log/lava-server/lava-logs.log, /var/log/lava-dispatcher/lava-slave.log)
Oh yes, there is something in lava-logs.log:
2018-04-30 11:00:52,046 DEBUG PING =
master (last message 20s ago)
2018-04-30 11:00:52,106 INFO master =
CANCEL(357)
2018-04-30 11:00:52,106 DEBUG [357] Unknown job, sending END 2018-04-30 11:00:52,107 ERROR [357] Unable to read 'description.yaml' 2018-04-30 11:00:52,107 ERROR [Errno 2] No such file or directory: '/var/lib/lava/dispatcher/slave/tmp/357/description.yaml' Traceback (most recent call last): File "/usr/bin/lava-slave", line 180, in description data = open(filename, 'r').read() IOError: [Errno 2] No such file or directory: '/var/lib/lava/dispatcher/slave/tmp/357/description.yaml' 2018-04-30 11:00:52,107 ERROR [357] lava-run crashed 2018-04-30 11:00:52,107 DEBUG master =
PONG(20)
2018-04-30 11:00:52,114 INFO master =
END_OK(357)
2018-04-30 11:00:52,114 DEBUG [357] Job END acked 2018-04-30 11:00:52,114 DEBUG [357] Removing /var/lib/lava/dispatcher/slave/tmp/357
Can you tell what this means?
3/ Which version of lava re you using?
2018.2
Regards
2018-04-30 10:14 GMT+02:00 Tim Jaacks <tim.jaacks@garz-fricke.com :
Hey everyone,
I still have this problem. Can anyone assist me? Rebooting the server does not help. :-/
Regards,
Tim
Hi Neil,
thanks for your quick help.
Hello everyone,
I added an email notification to a test job but forgot to configure
an SMTP server first.
There will likely be an exception logged in
/var/log/lava-server/lava-master.log
You will need to restart the lava-master
$ sudo service lava-master restart
Then check the logs.
My log says:
2018-04-24 14:22:08,025 INFO [POLL] Received a signal, leaving
2018-04-24 14:22:08,025 INFO [CLOSE] Closing the controler socket and dropping messages
2018-04-24 14:22:09,183 INFO [INIT] Dropping privileges
2018-04-24 14:22:09,184 DEBUG Switching to (lavaserver(114), lavaserver(118))
2018-04-24 14:22:09,184 INFO [INIT] Marking all workers as offline
2018-04-24 14:22:09,191 INFO [INIT] LAVA master has started.
2018-04-24 14:22:09,191 INFO [INIT] Using protocol version 3
2018-04-24 14:22:23,300 DEBUG lava-logs = PING(20)
2018-04-24 14:22:25,485 DEBUG A048.HAMBURG = PING(20)
2018-04-24 14:22:25,485 WARNING Unknown dispatcher <A048.HAMBURG (server crashed)
2018-04-24 14:22:29,195 INFO scheduling health checks:
2018-04-24 14:22:29,234 DEBUG - disabled on: lxc, qemu
2018-04-24 14:22:29,235 INFO scheduling jobs:
2018-04-24 14:22:29,236 DEBUG - beaglebone-black
2018-04-24 14:22:29,246 DEBUG - imx6guf
2018-04-24 14:22:29,255 DEBUG - lxc
2018-04-24 14:22:29,263 DEBUG - qemu
2018-04-24 14:22:29,284 INFO [357] CANCEL = A048.HAMBURG
2018-04-24 14:22:29,287 INFO [357] A048.HAMBURG = END (duplicated), skipping
What does this mean? FYI: Master and dispatcher are on the same machine. Can I restart the dispatcher somehow?
You don't need an SMTP server to be configured in LAVA but you do need
a working mail configuration on the master so that you can send email
to external addresses using the command line program, mail. The SMTP
settings in lava-server are an alternative. If you use the external
mail support, mail errors would remain in the email system of the master.
The job reported:
"JobError: Your job cannot terminate cleanly."
Afterwards nothing happened. The job was still running, even after
all timeouts had been passed, so I tried to cancel it. Now the job
remains in "Cancelling" state and I have no idea why and how to fix this. Any hints?
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT
ENGINEER Garz & Fricke GmbH Tempowerkring 2
21079 Hamburg
Direct: +49 40 791 899 - 55
Fax: +49 40 791899 - 39
tim.jaacks at garz-fricke.com
www.garz-fricke.com
SOLUTIONS THAT COMPLETE!
Sitz der Gesellschaft: D-21079 Hamburg
Registergericht: Amtsgericht Hamburg, HRB 60514
Geschäftsführer: Matthias Fricke, Manfred Garz
Lava-users mailing list
Lava-users at lists.linaro.org
--
Neil Williams
=============
neil.williams at linaro.org
-------------- next part -------------- An HTML attachment was
scrubbed...
URL:
<http://lists.linaro.org/pipermail/lava-users/attachments/20180424/e7a0
46b0/attachment.html
Mit freundlichen Grüßen / Best regards
Tim Jaacks
DEVELOPMENT ENGINEER
Garz & Fricke GmbH
Tempowerkring 2
21079 Hamburg
Direct: +49 40 791 899 - 55
Fax: +49 40 791899 - 39
tim.jaacks@garz-fricke.com
www.garz-fricke.com
SOLUTIONS THAT COMPLETE!
Sitz der Gesellschaft: D-21079 Hamburg
Registergericht: Amtsgericht Hamburg, HRB 60514
Geschäftsführer: Matthias Fricke, Manfred Garz _______________________________________________
Lava-users mailing list
Lava-users@lists.linaro.org
Lava-users mailing list
Lava-users@lists.linaro.org
--
Rémi Duraffort
LAVA Team
Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
-- Rémi Duraffort LAVA Team
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks@garz-fricke.com www.garz-fricke.com SOLUTIONS THAT COMPLETE!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz
Hello,
in 2018.4 a button is available in the web interface (job page). When a job is canceled, a "fail" button appear for the admins.
Regards
2018-05-07 9:43 GMT+02:00 Tim Jaacks tim.jaacks@garz-fricke.com:
If you are sure that the jobs is finished (the corresponding lava-run is
not running), you can force a job to finish by using "lava-server manage jobs fail <job_id
".
This will change the job and device state in the database directly. This
is useful if something went wrong and you are sure that lava-run has terminated.
Thanks Remi, that helped.
Regards
2018-04-30 13:04 GMT+02:00 Tim Jaacks <tim.jaacks@garz-fricke.com
:
Hi Remi,
thanks for your quick reply.
Hello,
rebooting the server will not fix anything as the status is saved into
the database (so kept between reboots).
1/ Are lava-master, lava-slave and lava-logs all running?
I think so, yes:
tim.jaacks@A048:~$ ps -ax | grep lava 637 ? Ssl 0:01 /usr/bin/python /usr/bin/lava-server manage
lava-publisher --level DEBUG
673 ? Ssl 0:41 /usr/bin/python /usr/bin/lava-server manage
lava-logs --level DEBUG
678 ? Ssl 1:26 /usr/bin/python /usr/bin/lava-slave --level
DEBUG --master tcp://localhost:5556 --socket-addr tcp://localhost:5555
824 ? S 0:00 /usr/bin/python /usr/bin/lava-coordinator
--loglevel=DEBUG
1017 ? S 9:30 /usr/bin/python /usr/sbin/lavapdu-runner
--loglevel=INFO
1018 ? S 0:51 /usr/bin/python /usr/sbin/lavapdu-listen
--loglevel=INFO
1473 ? Ss 0:00 postgres: 9.6/main: lavaserver lavaserver
::1(35670) idle
2476 ? Ss 1:19 gunicorn: master [lava_server.wsgi] 2482 ? S 0:12 gunicorn: worker [lava_server.wsgi] 2483 ? S 0:11 gunicorn: worker [lava_server.wsgi] 2484 ? S 0:13 gunicorn: worker [lava_server.wsgi] 2489 ? S 0:13 gunicorn: worker [lava_server.wsgi] 3077 ? Ssl 24:59 /usr/bin/python /usr/bin/lava-server manage
lava-master --level DEBUG
3079 ? Ss 8:13 postgres: 9.6/main: lavaserver lavaserver
::1(38208) idle
3418 pts/0 S+ 0:00 grep lava
2/ Do you see anything in the corresponding logs?
(/var/log/lava-server/lava-master.log, /var/log/lava-server/lava-logs.log, /var/log/lava-dispatcher/lava-slave.log)
Oh yes, there is something in lava-logs.log:
2018-04-30 11:00:52,046 DEBUG PING =
master (last message 20s ago)
2018-04-30 11:00:52,106 INFO master =
CANCEL(357)
2018-04-30 11:00:52,106 DEBUG [357] Unknown job, sending END 2018-04-30 11:00:52,107 ERROR [357] Unable to read 'description.yaml' 2018-04-30 11:00:52,107 ERROR [Errno 2] No such file or directory:
'/var/lib/lava/dispatcher/slave/tmp/357/description.yaml'
Traceback (most recent call last): File "/usr/bin/lava-slave", line 180, in description data = open(filename, 'r').read() IOError: [Errno 2] No such file or directory: '/var/lib/lava/dispatcher/
slave/tmp/357/description.yaml'
2018-04-30 11:00:52,107 ERROR [357] lava-run crashed 2018-04-30 11:00:52,107 DEBUG master =
PONG(20)
2018-04-30 11:00:52,114 INFO master =
END_OK(357)
2018-04-30 11:00:52,114 DEBUG [357] Job END acked 2018-04-30 11:00:52,114 DEBUG [357] Removing /var/lib/lava/dispatcher/
slave/tmp/357
Can you tell what this means?
3/ Which version of lava re you using?
2018.2
Regards
2018-04-30 10:14 GMT+02:00 Tim Jaacks <tim.jaacks@garz-fricke.com :
Hey everyone,
I still have this problem. Can anyone assist me? Rebooting the server
does not help. :-/
Regards,
Tim
Hi Neil,
thanks for your quick help.
Hello everyone,
I added an email notification to a test job but forgot to configure
an SMTP server first.
There will likely be an exception logged in
/var/log/lava-server/lava-master.log
You will need to restart the lava-master
$ sudo service lava-master restart
Then check the logs.
My log says:
2018-04-24 14:22:08,025 INFO [POLL] Received a signal, leaving
2018-04-24 14:22:08,025 INFO [CLOSE] Closing the controler socket and
dropping messages
2018-04-24 14:22:09,183 INFO [INIT] Dropping privileges
2018-04-24 14:22:09,184 DEBUG Switching to (lavaserver(114),
lavaserver(118))
2018-04-24 14:22:09,184 INFO [INIT] Marking all workers as offline
2018-04-24 14:22:09,191 INFO [INIT] LAVA master has started.
2018-04-24 14:22:09,191 INFO [INIT] Using protocol version 3
2018-04-24 14:22:23,300 DEBUG lava-logs = PING(20)
2018-04-24 14:22:25,485 DEBUG A048.HAMBURG = PING(20)
2018-04-24 14:22:25,485 WARNING Unknown dispatcher <A048.HAMBURG (server crashed)
2018-04-24 14:22:29,195 INFO scheduling health checks:
2018-04-24 14:22:29,234 DEBUG - disabled on: lxc, qemu
2018-04-24 14:22:29,235 INFO scheduling jobs:
2018-04-24 14:22:29,236 DEBUG - beaglebone-black
2018-04-24 14:22:29,246 DEBUG - imx6guf
2018-04-24 14:22:29,255 DEBUG - lxc
2018-04-24 14:22:29,263 DEBUG - qemu
2018-04-24 14:22:29,284 INFO [357] CANCEL = A048.HAMBURG
2018-04-24 14:22:29,287 INFO [357] A048.HAMBURG = END (duplicated), skipping
What does this mean? FYI: Master and dispatcher are on the same machine.
Can I restart the dispatcher somehow?
You don't need an SMTP server to be configured in LAVA but you do need
a working mail configuration on the master so that you can send email
to external addresses using the command line program, mail. The SMTP
settings in lava-server are an alternative. If you use the external
mail support, mail errors would remain in the email system of the master.
The job reported:
"JobError: Your job cannot terminate cleanly."
Afterwards nothing happened. The job was still running, even after
all timeouts had been passed, so I tried to cancel it. Now the job
remains in "Cancelling" state and I have no idea why and how to fix
this. Any hints?
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT
ENGINEER Garz & Fricke GmbH Tempowerkring 2
21079 Hamburg
Direct: +49 40 791 899 - 55
Fax: +49 40 791899 - 39
tim.jaacks at garz-fricke.com
www.garz-fricke.com
SOLUTIONS THAT COMPLETE!
Sitz der Gesellschaft: D-21079 Hamburg
Registergericht: Amtsgericht Hamburg, HRB 60514
Geschäftsführer: Matthias Fricke, Manfred Garz
Lava-users mailing list
Lava-users at lists.linaro.org
--
Neil Williams
=============
neil.williams at linaro.org
-------------- next part -------------- An HTML attachment was
scrubbed...
URL:
<http://lists.linaro.org/pipermail/lava-users/attachments/20180424/e7a0
46b0/attachment.html
Mit freundlichen Grüßen / Best regards
Tim Jaacks
DEVELOPMENT ENGINEER
Garz & Fricke GmbH
Tempowerkring 2
21079 Hamburg
Direct: +49 40 791 899 - 55
Fax: +49 40 791899 - 39
tim.jaacks@garz-fricke.com
www.garz-fricke.com
SOLUTIONS THAT COMPLETE!
Sitz der Gesellschaft: D-21079 Hamburg
Registergericht: Amtsgericht Hamburg, HRB 60514
Geschäftsführer: Matthias Fricke, Manfred Garz
Lava-users mailing list
Lava-users@lists.linaro.org
Lava-users mailing list
Lava-users@lists.linaro.org
--
Rémi Duraffort
LAVA Team
Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
-- Rémi Duraffort LAVA Team
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks@garz-fricke.com www.garz-fricke.com SOLUTIONS THAT COMPLETE!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz _______________________________________________ Lava-users mailing list Lava-users@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lava-users
lava-users@lists.lavasoftware.org
-
Neil Williams -
Remi Duraffort -
Tim Jaacks