Hey,
I've a device stuck in 'Running' state, but missing the usual appending of '#<123> <Job Name> [submitter]'. It does not accept new jobs and cycling it through it's health states does not change the state. We had to reboot our systems unplanned (and uncoordinated) today, so that might be the cause of the issue, but I would like to get my device back :D
Google surfaced an old, outdated FAQ in the lava sources [1], and a bug marked as solved in our version 2021.11 [2], but both didn't really don't work as expected.
Does somebody have some tips how to set/force the device state to 'Idle' manually?
Best regards, Olli
[1] https://git.lavasoftware.org/balikm/lava/-/blob/2015.01/doc/faq.rst [2] https://git.lavasoftware.org/lava/lava/-/issues/471
On Tue, May 31, 2022 at 9:04 PM Westermann, Oliver Oliver.Westermann@cognex.com wrote:
Hey,
I've a device stuck in 'Running' state, but missing the usual appending of '#<123> <Job Name> [submitter]'. It does not accept new jobs and cycling it through it's health states does not change the state. We had to reboot our systems unplanned (and uncoordinated) today, so that might be the cause of the issue, but I would like to get my device back :D
Google surfaced an old, outdated FAQ in the lava sources [1], and a bug marked as solved in our version 2021.11 [2], but both didn't really don't work as expected.
Does somebody have some tips how to set/force the device state to 'Idle' manually?
I put my bet on lava-dispatcher crashing somewhere along the way. Please try to restart lava-dispatcher and all services around it (webserver, ser2net, etc.). Once you've done this, you can mark the device "idle" in the admin interface. This should help.
Best Regards, Milosz
Best regards, Olli
[1] https://git.lavasoftware.org/balikm/lava/-/blob/2015.01/doc/faq.rst [2] https://git.lavasoftware.org/lava/lava/-/issues/471 _______________________________________________ Lava-users mailing list -- lava-users@lists.lavasoftware.org To unsubscribe send an email to lava-users-leave@lists.lavasoftware.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
Hey,
From: Milosz Wasilewski milosz.wasilewski@foundries.io
On Tue, May 31, 2022 at 9:04 PM Westermann, Oliver Oliver.Westermann@cognex.com wrote:
Hey,
I've a device stuck in 'Running' state, but missing the usual appending of '#<123> <Job Name> [submitter]'.
Does somebody have some tips how to set/force the device state to 'Idle' manually?
I put my bet on lava-dispatcher crashing somewhere along the way. Please try to restart lava-dispatcher and all services around it (webserver, ser2net, etc.).
Did so before my mail request. We're running the docker-compose setup and I've already update (to 2022.05) and restarted (and rebuild) the containers with no success.
Once you've done this, you can mark the device "idle" in the admin interface. This should help.
Can you give me some guidance here? The admin interface (as well as the lava-server cli) allow me to set the health (good, bad, maintenance, retired), but not the state.
Unrelated sidenote: Rebuilding the docker containers failed me. The dispatcher requires a file from schneider electric [1], which is currently not available for me. I found the file on Remis Github [2] and fixed my Dockerfile locally, but I will try to create a Issue in the gitlab.
Best regards, Olli
[1] https://git.lavasoftware.org/lava/pkg/docker-compose/-/blob/master/dispatche... [2] https://github.com/ci-box/ci-box-lava-worker/blob/master/powernet428.mib
On Thu, Jun 2, 2022 at 10:36 AM Westermann, Oliver Oliver.Westermann@cognex.com wrote:
Hey,
From: Milosz Wasilewski milosz.wasilewski@foundries.io
On Tue, May 31, 2022 at 9:04 PM Westermann, Oliver Oliver.Westermann@cognex.com wrote:
Hey,
I've a device stuck in 'Running' state, but missing the usual appending of '#<123> <Job Name> [submitter]'.
Does somebody have some tips how to set/force the device state to 'Idle' manually?
I put my bet on lava-dispatcher crashing somewhere along the way. Please try to restart lava-dispatcher and all services around it (webserver, ser2net, etc.).
Did so before my mail request. We're running the docker-compose setup and I've already update (to 2022.05) and restarted (and rebuild) the containers with no success.
That's weird. It should fix the issue
Once you've done this, you can mark the device "idle" in the admin interface. This should help.
Can you give me some guidance here? The admin interface (as well as the lava-server cli) allow me to set the health (good, bad, maintenance, retired), but not the state.
My bad, you're absolutely right. The state field doesn't seem to be editable. You might need to change it in the DB. I've no idea how it got to this point. In my case when dispatcher is restarted the device status usually fixes itself. One thing to consider is deleting the offending TestJob from DB. This might help (I didn't look at the code to confirm it)
Unrelated sidenote: Rebuilding the docker containers failed me. The dispatcher requires a file from schneider electric [1], which is currently not available for me. I found the file on Remis Github [2] and fixed my Dockerfile locally, but I will try to create a Issue in the gitlab.
It's MIB for APC PDU (or some other Schneider equipment). It's only required if you're using SNMP to talk to PDU with the name instead of it's ASN.1 number. Linaro's lab-scripts [1] make use of this MIB.
[1] https://git.linaro.org/lava/lava-lab.git/tree/shared/lab-scripts/snmp_pdu_co...
Best Regards, Milosz
Best regards, Olli
[1] https://git.lavasoftware.org/lava/pkg/docker-compose/-/blob/master/dispatche... [2] https://github.com/ci-box/ci-box-lava-worker/blob/master/powernet428.mib
lava-users@lists.lavasoftware.org
-
Milosz Wasilewski -
Westermann, Oliver