After getting stats on my setup robustness, the step forward is have a complete view on the lava errors we meet in incomplete jobs.
From what I see in incomplete jobs, my intention is to query on test suite lava and the name "job".
In the query builder, if I use test suite as condition model, I can't use the job field name.
Do you have any advice on how to proceed?
Denis
On Wed, 30 Jan 2019 at 10:11, Denis HUMEAU denis.humeau@st.com wrote:
After getting stats on my setup robustness, the step forward is have a complete view on the lava errors we meet in incomplete jobs.
Query isn't best suited for this - all you'll get are the testcases / Job IDs which match, not the reason for the failure. There is already a table for Recent Job Errors: http://localhost/scheduler/joberrors
The Query|Charts support in LAVA is deliberately simple and limited. Admins can use lava-server manage operations to get specific information and there are XMLRPC and REST API calls which can be made too. Quite quickly, other reporting support will be needed in lots of labs but each set of queries is different and custom reporting becomes necessary.
From what I see in incomplete jobs, my intention is to query on test suite lava and the name “job”.
I got limited use from: http://localhost/results/query/+custom?entity=testsuite&conditions=tests...
More instructive, I find, is to use notifications in health checks and/or functional tests:
https://git.lavasoftware.org/lava/functional-tests/blob/master/release/cubie...
The problem with looking at all incomplete test jobs is that you need to filter out genuine test failures (e.g. kernel didn't boot) from the Infrastructure errors or bugs. That's what the Recent Job Errors table does.
For XMLRPC you could start with http://localhost/api/help/#scheduler.jobs.list
`scheduler.jobs.list` ( `state=None`, `health=None`, `start=0`, `limit=25`, `since=None`, `verbose=False` )
Crucially, this call includes the "error_msg" & "error_type" in the first data set, if verbose is set.
Note: the reason why these are not typically included is that retrieving this data involves a lot more SQL queries under the hood. That's also why it is not possible for a Query to dig into that data.
Additionally, lavacli can do the work of querying the scheduler jobs list.
$ lavacli -i staging jobs list --health INCOMPLETE --state FINISHED --limit 5 --verbose --yaml - {actual_device: staging-x15-01, description: x-15 tftp nfs OE, device_type: x15, end_time: '2019-01-29 20:42:16.329289+00:00', error_msg: auto-login-action timed out after 1768 seconds, error_type: Job, health: Incomplete, id: 248207, start_time: '2019-01-29 20:11:30.376908+00:00', state: Finished, submitter: neil.williams@linaro.org} - {actual_device: staging-x15-01, description: x-15 tftp nfs OE, device_type: x15, end_time: '2019-01-29 20:08:38.065813+00:00', error_msg: auto-login-action timed out after 522 seconds, error_type: Job, health: Incomplete, id: 248206, start_time: '2019-01-29 19:58:28.706303+00:00', state: Finished, submitter: neil.williams@linaro.org} - {actual_device: staging-hi960-hikey-02, description: HiKey 960 Android boot test, device_type: hi960-hikey, end_time: '2019-01-28 12:50:33.255416+00:00', error_msg: /usr/local/lab-scripts/cbrxd_hub_control -i DJ008WLX -m sync -u 15 failed, error_type: Infrastructure, health: Incomplete, id: 248142, start_time: '2019-01-28 12:47:44.897788+00:00', state: Finished, submitter: lava-health} - {actual_device: staging-black04, description: beaglebone-black standard NFS health check, device_type: beaglebone-black, end_time: '2019-01-26 19:03:12.393021+00:00', error_msg: 'matched a bootloader error message: ''Retry count exceeded'' (4)', error_type: Infrastructure, health: Incomplete, id: 248104, start_time: '2019-01-26 19:01:51.542197+00:00', state: Finished, submitter: lava-health} - {actual_device: staging-hi960-hikey-02, description: HiKey 960 Android boot test, device_type: hi960-hikey, end_time: '2019-01-25 16:31:58.165753+00:00', error_msg: /usr/local/lab-scripts/cbrxd_hub_control -i DJ008WLX -m sync -u 15 failed, error_type: Infrastructure, health: Incomplete, id: 248065, start_time: '2019-01-25 16:27:59.082922+00:00', state: Finished, submitter: lava-health}
On Wed, 30 Jan 2019 at 12:48, Neil Williams neil.williams@linaro.org wrote:
On Wed, 30 Jan 2019 at 10:11, Denis HUMEAU denis.humeau@st.com wrote:
After getting stats on my setup robustness, the step forward is have a complete view on the lava errors we meet in incomplete jobs.
Query isn't best suited for this - all you'll get are the testcases / Job IDs which match, not the reason for the failure. There is already a table for Recent Job Errors: http://localhost/scheduler/joberrors
The Query|Charts support in LAVA is deliberately simple and limited. Admins can use lava-server manage operations to get specific information and there are XMLRPC and REST API calls which can be made too. Quite quickly, other reporting support will be needed in lots of labs but each set of queries is different and custom reporting becomes necessary.
From what I see in incomplete jobs, my intention is to query on test suite lava and the name “job”.
I got limited use from: http://localhost/results/query/+custom?entity=testsuite&conditions=tests...
More instructive, I find, is to use notifications in health checks and/or functional tests:
https://git.lavasoftware.org/lava/functional-tests/blob/master/release/cubie...
The problem with looking at all incomplete test jobs is that you need to filter out genuine test failures (e.g. kernel didn't boot) from the Infrastructure errors or bugs. That's what the Recent Job Errors table does.
For XMLRPC you could start with http://localhost/api/help/#scheduler.jobs.list
`scheduler.jobs.list` ( `state=None`, `health=None`, `start=0`, `limit=25`, `since=None`, `verbose=False` )
Crucially, this call includes the "error_msg" & "error_type" in the first data set, if verbose is set.
Note: the reason why these are not typically included is that retrieving this data involves a lot more SQL queries under the hood. That's also why it is not possible for a Query to dig into that data.
--
Neil Williams
neil.williams@linaro.org http://www.linux.codehelp.co.uk/
Hello,
It seems that the option --verbose is no longer present. How can I get a view on error_msg via lavacli?
Best regards,
Denis
-----Original Message----- From: Neil Williams neil.williams@linaro.org Sent: mercredi 30 janvier 2019 13:54 To: Denis HUMEAU denis.humeau@st.com Cc: lava-users lava-users@lists.lavasoftware.org Subject: Re: [Lava-users] query to retrieve Lava errors
Additionally, lavacli can do the work of querying the scheduler jobs list.
$ lavacli -i staging jobs list --health INCOMPLETE --state FINISHED --limit 5 --verbose --yaml - {actual_device: staging-x15-01, description: x-15 tftp nfs OE, device_type: x15, end_time: '2019-01-29 20:42:16.329289+00:00', error_msg: auto-login-action timed out after 1768 seconds, error_type: Job, health: Incomplete, id: 248207, start_time: '2019-01-29 20:11:30.376908+00:00', state: Finished, submitter: neil.williams@linaro.org} - {actual_device: staging-x15-01, description: x-15 tftp nfs OE, device_type: x15, end_time: '2019-01-29 20:08:38.065813+00:00', error_msg: auto-login-action timed out after 522 seconds, error_type: Job, health: Incomplete, id: 248206, start_time: '2019-01-29 19:58:28.706303+00:00', state: Finished, submitter: neil.williams@linaro.org} - {actual_device: staging-hi960-hikey-02, description: HiKey 960 Android boot test, device_type: hi960-hikey, end_time: '2019-01-28 12:50:33.255416+00:00', error_msg: /usr/local/lab-scripts/cbrxd_hub_control -i DJ008WLX -m sync -u 15 failed, error_type: Infrastructure, health: Incomplete, id: 248142, start_time: '2019-01-28 12:47:44.897788+00:00', state: Finished, submitter: lava-health} - {actual_device: staging-black04, description: beaglebone-black standard NFS health check, device_type: beaglebone-black, end_time: '2019-01-26 19:03:12.393021+00:00', error_msg: 'matched a bootloader error message: ''Retry count exceeded'' (4)', error_type: Infrastructure, health: Incomplete, id: 248104, start_time: '2019-01-26 19:01:51.542197+00:00', state: Finished, submitter: lava-health} - {actual_device: staging-hi960-hikey-02, description: HiKey 960 Android boot test, device_type: hi960-hikey, end_time: '2019-01-25 16:31:58.165753+00:00', error_msg: /usr/local/lab-scripts/cbrxd_hub_control -i DJ008WLX -m sync -u 15 failed, error_type: Infrastructure, health: Incomplete, id: 248065, start_time: '2019-01-25 16:27:59.082922+00:00', state: Finished, submitter: lava-health}
On Wed, 30 Jan 2019 at 12:48, Neil Williams neil.williams@linaro.org wrote:
On Wed, 30 Jan 2019 at 10:11, Denis HUMEAU denis.humeau@st.com wrote:
After getting stats on my setup robustness, the step forward is have a complete view on the lava errors we meet in incomplete jobs.
Query isn't best suited for this - all you'll get are the testcases / Job IDs which match, not the reason for the failure. There is already a table for Recent Job Errors: http://localhost/scheduler/joberrors
The Query|Charts support in LAVA is deliberately simple and limited. Admins can use lava-server manage operations to get specific information and there are XMLRPC and REST API calls which can be made too. Quite quickly, other reporting support will be needed in lots of labs but each set of queries is different and custom reporting becomes necessary.
From what I see in incomplete jobs, my intention is to query on test suite lava and the name “job”.
I got limited use from: http://localhost/results/query/+custom?entity=testsuite&conditions=tes tsuite__name__exact__lava,testcase__result__exact__Test%20failed,testc ase__name__exact__job
More instructive, I find, is to use notifications in health checks and/or functional tests:
https://git.lavasoftware.org/lava/functional-tests/blob/master/release /cubietruck-ssh-guest.yaml#L17
The problem with looking at all incomplete test jobs is that you need to filter out genuine test failures (e.g. kernel didn't boot) from the Infrastructure errors or bugs. That's what the Recent Job Errors table does.
For XMLRPC you could start with http://localhost/api/help/#scheduler.jobs.list
`scheduler.jobs.list` ( `state=None`, `health=None`, `start=0`, `limit=25`, `since=None`, `verbose=False` )
Crucially, this call includes the "error_msg" & "error_type" in the first data set, if verbose is set.
Note: the reason why these are not typically included is that retrieving this data involves a lot more SQL queries under the hood. That's also why it is not possible for a Query to dig into that data.
--
Neil Williams
neil.williams@linaro.org http://www.linux.codehelp.co.uk/
Which version of lavacli are yo using?
I just tried locally (also playing with jq for the filtering):
lavacli -i staging jobs list --health INCOMPLETE --state FINISHED --limit 5 --verbose --json | jq '.[].error_msg' "lava-test-shell timed out after 3149 seconds" "fastboot-flash-action timed out after 2292 seconds" "fastboot-flash-action timed out after 2290 seconds" "Connection closed" "fastboot-flash-action timed out after 7075 seconds"
lavacli --version 0.9.5 (also tested with 0.9.6)
Le ven. 14 juin 2019 à 12:16, Denis HUMEAU denis.humeau@st.com a écrit :
Hello,
It seems that the option --verbose is no longer present. How can I get a view on error_msg via lavacli?
Best regards,
Denis
-----Original Message----- From: Neil Williams neil.williams@linaro.org Sent: mercredi 30 janvier 2019 13:54 To: Denis HUMEAU denis.humeau@st.com Cc: lava-users lava-users@lists.lavasoftware.org Subject: Re: [Lava-users] query to retrieve Lava errors
Additionally, lavacli can do the work of querying the scheduler jobs list.
$ lavacli -i staging jobs list --health INCOMPLETE --state FINISHED --limit 5 --verbose --yaml
- {actual_device: staging-x15-01, description: x-15 tftp nfs OE,
device_type: x15, end_time: '2019-01-29 20:42:16.329289+00:00', error_msg: auto-login-action timed out after 1768 seconds, error_type: Job, health: Incomplete, id: 248207, start_time: '2019-01-29 20:11:30.376908+00:00', state: Finished, submitter: neil.williams@linaro.org}
- {actual_device: staging-x15-01, description: x-15 tftp nfs OE,
device_type: x15, end_time: '2019-01-29 20:08:38.065813+00:00', error_msg: auto-login-action timed out after 522 seconds, error_type: Job, health: Incomplete, id: 248206, start_time: '2019-01-29 19:58:28.706303+00:00', state: Finished, submitter: neil.williams@linaro.org}
- {actual_device: staging-hi960-hikey-02, description: HiKey 960 Android
boot test, device_type: hi960-hikey, end_time: '2019-01-28 12:50:33.255416+00:00', error_msg: /usr/local/lab-scripts/cbrxd_hub_control -i DJ008WLX -m sync -u 15 failed, error_type: Infrastructure, health: Incomplete, id: 248142, start_time: '2019-01-28 12:47:44.897788+00:00', state: Finished, submitter: lava-health}
- {actual_device: staging-black04, description: beaglebone-black standard
NFS health check, device_type: beaglebone-black, end_time: '2019-01-26 19:03:12.393021+00:00', error_msg: 'matched a bootloader error message: ''Retry count exceeded'' (4)', error_type: Infrastructure, health: Incomplete, id: 248104, start_time: '2019-01-26 19:01:51.542197+00:00', state: Finished, submitter: lava-health}
- {actual_device: staging-hi960-hikey-02, description: HiKey 960 Android
boot test, device_type: hi960-hikey, end_time: '2019-01-25 16:31:58.165753+00:00', error_msg: /usr/local/lab-scripts/cbrxd_hub_control -i DJ008WLX -m sync -u 15 failed, error_type: Infrastructure, health: Incomplete, id: 248065, start_time: '2019-01-25 16:27:59.082922+00:00', state: Finished, submitter: lava-health}
On Wed, 30 Jan 2019 at 12:48, Neil Williams neil.williams@linaro.org wrote:
On Wed, 30 Jan 2019 at 10:11, Denis HUMEAU denis.humeau@st.com wrote:
After getting stats on my setup robustness, the step forward is have a
complete view on the lava errors we meet in incomplete jobs.
Query isn't best suited for this - all you'll get are the testcases / Job IDs which match, not the reason for the failure. There is already a table for Recent Job Errors: http://localhost/scheduler/joberrors
The Query|Charts support in LAVA is deliberately simple and limited. Admins can use lava-server manage operations to get specific information and there are XMLRPC and REST API calls which can be made too. Quite quickly, other reporting support will be needed in lots of labs but each set of queries is different and custom reporting becomes necessary.
From what I see in incomplete jobs, my intention is to query on test
suite lava and the name “job”.
I got limited use from: http://localhost/results/query/+custom?entity=testsuite&conditions=tes tsuite__name__exact__lava,testcase__result__exact__Test%20failed,testc ase__name__exact__job
More instructive, I find, is to use notifications in health checks and/or functional tests:
https://git.lavasoftware.org/lava/functional-tests/blob/master/release /cubietruck-ssh-guest.yaml#L17
The problem with looking at all incomplete test jobs is that you need to filter out genuine test failures (e.g. kernel didn't boot) from the Infrastructure errors or bugs. That's what the Recent Job Errors table does.
For XMLRPC you could start with http://localhost/api/help/#scheduler.jobs.list
`scheduler.jobs.list` ( `state=None`, `health=None`, `start=0`, `limit=25`, `since=None`, `verbose=False` )
Crucially, this call includes the "error_msg" & "error_type" in the first data set, if verbose is set.
Note: the reason why these are not typically included is that retrieving this data involves a lot more SQL queries under the hood. That's also why it is not possible for a Query to dig into that data.
--
Neil Williams
neil.williams@linaro.org http://www.linux.codehelp.co.uk/
--
Neil Williams
neil.williams@linaro.org http://www.linux.codehelp.co.uk/ _______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
0.9.2. I’ll switch to the latest
Thanks Rémi
From: Remi Duraffort remi.duraffort@linaro.org Sent: vendredi 14 juin 2019 12:28 To: Denis HUMEAU denis.humeau@st.com Cc: Neil Williams neil.williams@linaro.org; lava-users lava-users@lists.lavasoftware.org Subject: Re: [Lava-users] query to retrieve Lava errors
Which version of lavacli are yo using?
I just tried locally (also playing with jq for the filtering):
lavacli -i staging jobs list --health INCOMPLETE --state FINISHED --limit 5 --verbose --json | jq '.[].error_msg' "lava-test-shell timed out after 3149 seconds" "fastboot-flash-action timed out after 2292 seconds" "fastboot-flash-action timed out after 2290 seconds" "Connection closed" "fastboot-flash-action timed out after 7075 seconds"
lavacli --version 0.9.5 (also tested with 0.9.6)
Le ven. 14 juin 2019 à 12:16, Denis HUMEAU <denis.humeau@st.commailto:denis.humeau@st.com> a écrit : Hello,
It seems that the option --verbose is no longer present. How can I get a view on error_msg via lavacli?
Best regards,
Denis
-----Original Message----- From: Neil Williams <neil.williams@linaro.orgmailto:neil.williams@linaro.org> Sent: mercredi 30 janvier 2019 13:54 To: Denis HUMEAU <denis.humeau@st.commailto:denis.humeau@st.com> Cc: lava-users <lava-users@lists.lavasoftware.orgmailto:lava-users@lists.lavasoftware.org> Subject: Re: [Lava-users] query to retrieve Lava errors
Additionally, lavacli can do the work of querying the scheduler jobs list.
$ lavacli -i staging jobs list --health INCOMPLETE --state FINISHED --limit 5 --verbose --yaml - {actual_device: staging-x15-01, description: x-15 tftp nfs OE, device_type: x15, end_time: '2019-01-29 20:42:16.329289+00:00', error_msg: auto-login-action timed out after 1768 seconds, error_type: Job, health: Incomplete, id: 248207, start_time: '2019-01-29 20:11:30.376908+00:00', state: Finished, submitter: neil.williams@linaro.orgmailto:neil.williams@linaro.org} - {actual_device: staging-x15-01, description: x-15 tftp nfs OE, device_type: x15, end_time: '2019-01-29 20:08:38.065813+00:00', error_msg: auto-login-action timed out after 522 seconds, error_type: Job, health: Incomplete, id: 248206, start_time: '2019-01-29 19:58:28.706303+00:00', state: Finished, submitter: neil.williams@linaro.orgmailto:neil.williams@linaro.org} - {actual_device: staging-hi960-hikey-02, description: HiKey 960 Android boot test, device_type: hi960-hikey, end_time: '2019-01-28 12:50:33.255416+00:00', error_msg: /usr/local/lab-scripts/cbrxd_hub_control -i DJ008WLX -m sync -u 15 failed, error_type: Infrastructure, health: Incomplete, id: 248142, start_time: '2019-01-28 12:47:44.897788+00:00', state: Finished, submitter: lava-health} - {actual_device: staging-black04, description: beaglebone-black standard NFS health check, device_type: beaglebone-black, end_time: '2019-01-26 19:03:12.393021+00:00', error_msg: 'matched a bootloader error message: ''Retry count exceeded'' (4)', error_type: Infrastructure, health: Incomplete, id: 248104, start_time: '2019-01-26 19:01:51.542197+00:00', state: Finished, submitter: lava-health} - {actual_device: staging-hi960-hikey-02, description: HiKey 960 Android boot test, device_type: hi960-hikey, end_time: '2019-01-25 16:31:58.165753+00:00', error_msg: /usr/local/lab-scripts/cbrxd_hub_control -i DJ008WLX -m sync -u 15 failed, error_type: Infrastructure, health: Incomplete, id: 248065, start_time: '2019-01-25 16:27:59.082922+00:00', state: Finished, submitter: lava-health}
On Wed, 30 Jan 2019 at 12:48, Neil Williams <neil.williams@linaro.orgmailto:neil.williams@linaro.org> wrote:
On Wed, 30 Jan 2019 at 10:11, Denis HUMEAU <denis.humeau@st.commailto:denis.humeau@st.com> wrote:
After getting stats on my setup robustness, the step forward is have a complete view on the lava errors we meet in incomplete jobs.
Query isn't best suited for this - all you'll get are the testcases / Job IDs which match, not the reason for the failure. There is already a table for Recent Job Errors: http://localhost/scheduler/joberrors
The Query|Charts support in LAVA is deliberately simple and limited. Admins can use lava-server manage operations to get specific information and there are XMLRPC and REST API calls which can be made too. Quite quickly, other reporting support will be needed in lots of labs but each set of queries is different and custom reporting becomes necessary.
From what I see in incomplete jobs, my intention is to query on test suite lava and the name “job”.
I got limited use from: http://localhost/results/query/+custom?entity=testsuite&conditions=tes tsuite__name__exact__lava,testcase__result__exact__Test%20failed,testc ase__name__exact__job
More instructive, I find, is to use notifications in health checks and/or functional tests:
https://git.lavasoftware.org/lava/functional-tests/blob/master/release /cubietruck-ssh-guest.yaml#L17
The problem with looking at all incomplete test jobs is that you need to filter out genuine test failures (e.g. kernel didn't boot) from the Infrastructure errors or bugs. That's what the Recent Job Errors table does.
For XMLRPC you could start with http://localhost/api/help/#scheduler.jobs.list
`scheduler.jobs.list` ( `state=None`, `health=None`, `start=0`, `limit=25`, `since=None`, `verbose=False` )
Crucially, this call includes the "error_msg" & "error_type" in the first data set, if verbose is set.
Note: the reason why these are not typically included is that retrieving this data involves a lot more SQL queries under the hood. That's also why it is not possible for a Query to dig into that data.
--
Neil Williams
neil.williams@linaro.orgmailto:neil.williams@linaro.org http://www.linux.codehelp.co.uk/
--
Neil Williams ============= neil.williams@linaro.orgmailto:neil.williams@linaro.org http://www.linux.codehelp.co.uk/ _______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.orgmailto:Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
-- Rémi Duraffort LAVA Team, Linaro
lava-users@lists.lavasoftware.org
-
Denis HUMEAU -
Neil Williams -
Remi Duraffort