Hi Team,
I was running the lava jobs and I found a difference in the status after job completion with failures. There were failures in i) The interactive test and lava overlay test [cases] but still the status of the job shows as "COMPLETE". ii) I tried an action failure retry, when action failure for the first time, it shows the result as failure but job status as complete. The retry- failure status should be ignored if it fails for the first time. I convert the test results into JUNIT using rest api. In the junit result it displays failure for the test case. My question is, How can we customize the status of the job or ignore the test result in case of failure-retry? Any Advice please.
Best Regards *Pavan Kumar*
Pawan,
I think you're confusing 2 different things. Job status only says if all actions in the job completed successfully. For example if your test script runs for too long, lava will call a timeout and end the job as 'incomplete'. Similarly if your infrastructure doesn't work properly (power_on command doesn't power the device on, etc). On the other hand test results can have 4 values: pass, fail, skip, unknown. The last one is only telling that something went wrong with data collection.
To sum up, failed test doesn't mean that the test job is incomplete. These are statuses of 2 different things.
Best Regards, Milosz
On Wed, Feb 7, 2024 at 4:08 AM pawanKumar pawan4749@gmail.com wrote:
Hi Team,
I was running the lava jobs and I found a difference in the status after job completion with failures. There were failures in i) The interactive test and lava overlay test [cases] but still the status of the job shows as "COMPLETE". ii) I tried an action failure retry, when action failure for the first time, it shows the result as failure but job status as complete. The retry- failure status should be ignored if it fails for the first time. I convert the test results into JUNIT using rest api. In the junit result it displays failure for the test case. My question is, How can we customize the status of the job or ignore the test result in case of failure-retry? Any Advice please.
Best Regards Pavan Kumar _______________________________________________ lava-devel mailing list -- lava-devel@lists.lavasoftware.org To unsubscribe send an email to lava-devel-leave@lists.lavasoftware.org
Hi Milosz
Yes, I understand that.
But in my case I'm exporting results to bamboo using rest api. My issue is about failure retry case, if the action was failure for the first time it shows result as fail. Even though it is pass at second time, Rest API takes the first failure and displays as failure test result in bamboo.
In this case implementation of failure retry would be not useful. So, is there a way to override/ignore the first failure case.
Thanks.
On Wed, 7 Feb, 2024, 2:34 pm Milosz Wasilewski, < milosz.wasilewski@foundries.io> wrote:
Pawan,
I think you're confusing 2 different things. Job status only says if all actions in the job completed successfully. For example if your test script runs for too long, lava will call a timeout and end the job as 'incomplete'. Similarly if your infrastructure doesn't work properly (power_on command doesn't power the device on, etc). On the other hand test results can have 4 values: pass, fail, skip, unknown. The last one is only telling that something went wrong with data collection.
To sum up, failed test doesn't mean that the test job is incomplete. These are statuses of 2 different things.
Best Regards, Milosz
On Wed, Feb 7, 2024 at 4:08 AM pawanKumar pawan4749@gmail.com wrote:
Hi Team,
I was running the lava jobs and I found a difference in the status after
job completion with failures. There were failures in
i) The interactive test and lava overlay test [cases] but still the
status of the job shows as "COMPLETE".
ii) I tried an action failure retry, when action failure for the first
time, it shows the result as failure but job status as complete. The retry- failure status should be ignored if it fails for the first time. I convert the test results into JUNIT using rest api. In the junit result it displays failure for the test case.
My question is, How can we customize the status of the job or ignore the
test result in case of failure-retry?
Any Advice please.
Best Regards Pavan Kumar _______________________________________________ lava-devel mailing list -- lava-devel@lists.lavasoftware.org To unsubscribe send an email to lava-devel-leave@lists.lavasoftware.org
On Thu, Feb 8, 2024 at 3:15 AM pawanKumar pawan4749@gmail.com wrote:
Hi Milosz
Yes, I understand that.
But in my case I'm exporting results to bamboo using rest api. My issue is about failure retry case, if the action was failure for the first time it shows result as fail. Even though it is pass at second time, Rest API takes the first failure and displays as failure test result in bamboo.
In this case implementation of failure retry would be not useful. So, is there a way to override/ignore the first failure case.
Sounds like a bug, do you have an example job with this behaviour?
Best Regards, Milosz
Thanks.
On Wed, 7 Feb, 2024, 2:34 pm Milosz Wasilewski, milosz.wasilewski@foundries.io wrote:
Pawan,
I think you're confusing 2 different things. Job status only says if all actions in the job completed successfully. For example if your test script runs for too long, lava will call a timeout and end the job as 'incomplete'. Similarly if your infrastructure doesn't work properly (power_on command doesn't power the device on, etc). On the other hand test results can have 4 values: pass, fail, skip, unknown. The last one is only telling that something went wrong with data collection.
To sum up, failed test doesn't mean that the test job is incomplete. These are statuses of 2 different things.
Best Regards, Milosz
On Wed, Feb 7, 2024 at 4:08 AM pawanKumar pawan4749@gmail.com wrote:
Hi Team,
I was running the lava jobs and I found a difference in the status after job completion with failures. There were failures in i) The interactive test and lava overlay test [cases] but still the status of the job shows as "COMPLETE". ii) I tried an action failure retry, when action failure for the first time, it shows the result as failure but job status as complete. The retry- failure status should be ignored if it fails for the first time. I convert the test results into JUNIT using rest api. In the junit result it displays failure for the test case. My question is, How can we customize the status of the job or ignore the test result in case of failure-retry? Any Advice please.
Best Regards Pavan Kumar _______________________________________________ lava-devel mailing list -- lava-devel@lists.lavasoftware.org To unsubscribe send an email to lava-devel-leave@lists.lavasoftware.org
Here is the Job definition I'm using and the debug log I have pasted at https://pastebin.com/2s2eTkxj Please let me know if there are any difficulties in checking the log.
*Job Definition:* device_type: ADT-HYBRID job_name: Flash and Run FCT Test timeouts: job: minutes: 50 action: minutes: 20 connection: minutes: 10 visibility: public actions: - command: name: relay_pwr_on timeout: minutes: 1 - deploy: failure_retry: 3 to: flasher images: package: url: https:// - boot: method: u-boot commands: - setenv factorymode 1 - boot failure_retry: 3 auto_login: login_prompt: 'login: ' username: root password_prompt: 'Password:' password: root login_commands: - ifconfig eth0 - sysinfo prompts: - 'root@hon-grip' - 'root@GRIP' transfer_overlay: download_command: wget -S unpack_command: tar -C / -xzf - test: failure_retry: 3 timeout: minutes: 20 definitions: - repository: ssh://git@github.com/gop/lava-tests.git from: git path: lava-test/hybrid-test.yaml name: test
Best Regards *Pavan Kumar*
On Thu, Feb 8, 2024 at 3:12 PM Milosz Wasilewski < milosz.wasilewski@foundries.io> wrote:
On Thu, Feb 8, 2024 at 3:15 AM pawanKumar pawan4749@gmail.com wrote:
Hi Milosz
Yes, I understand that.
But in my case I'm exporting results to bamboo using rest api. My issue
is about failure retry case, if the action was failure for the first time it shows result as fail. Even though it is pass at second time, Rest API takes the first failure and displays as failure test result in bamboo.
In this case implementation of failure retry would be not useful. So, is
there a way to override/ignore the first failure case.
Sounds like a bug, do you have an example job with this behaviour?
Best Regards, Milosz
Thanks.
On Wed, 7 Feb, 2024, 2:34 pm Milosz Wasilewski, <
milosz.wasilewski@foundries.io> wrote:
Pawan,
I think you're confusing 2 different things. Job status only says if all actions in the job completed successfully. For example if your test script runs for too long, lava will call a timeout and end the job as 'incomplete'. Similarly if your infrastructure doesn't work properly (power_on command doesn't power the device on, etc). On the other hand test results can have 4 values: pass, fail, skip, unknown. The last one is only telling that something went wrong with data collection.
To sum up, failed test doesn't mean that the test job is incomplete. These are statuses of 2 different things.
Best Regards, Milosz
On Wed, Feb 7, 2024 at 4:08 AM pawanKumar pawan4749@gmail.com wrote:
Hi Team,
I was running the lava jobs and I found a difference in the status
after job completion with failures. There were failures in
i) The interactive test and lava overlay test [cases] but still the
status of the job shows as "COMPLETE".
ii) I tried an action failure retry, when action failure for the
first time, it shows the result as failure but job status as complete. The retry- failure status should be ignored if it fails for the first time. I convert the test results into JUNIT using rest api. In the junit result it displays failure for the test case.
My question is, How can we customize the status of the job or ignore
the test result in case of failure-retry?
Any Advice please.
Best Regards Pavan Kumar _______________________________________________ lava-devel mailing list -- lava-devel@lists.lavasoftware.org To unsubscribe send an email to
lava-devel-leave@lists.lavasoftware.org
Pawan,
I'm confused. You're complaining about REST API. In such case the test definition doesn't help much. I'm not going to run it (the repository is private anyway). It would help if I have access to REST API with job results that you're complaining about. Otherwise I can't help, sorry.
Best Regards, Milosz
On Thu, Feb 8, 2024 at 3:10 PM pawanKumar pawan4749@gmail.com wrote:
Here is the Job definition I'm using and the debug log I have pasted at https://pastebin.com/2s2eTkxj Please let me know if there are any difficulties in checking the log.
Job Definition: device_type: ADT-HYBRID job_name: Flash and Run FCT Test timeouts: job: minutes: 50 action: minutes: 20 connection: minutes: 10 visibility: public actions:
- command: name: relay_pwr_on timeout: minutes: 1
- deploy: failure_retry: 3 to: flasher images: package: url: https://
- boot: method: u-boot commands: - setenv factorymode 1 - boot failure_retry: 3 auto_login: login_prompt: 'login: ' username: root password_prompt: 'Password:' password: root login_commands: - ifconfig eth0 - sysinfo prompts:
transfer_overlay: download_command: wget -S unpack_command: tar -C / -xzf
- 'root@hon-grip'
- 'root@GRIP'
- test: failure_retry: 3 timeout: minutes: 20 definitions:
- repository: ssh://git@github.com/gop/lava-tests.git from: git path: lava-test/hybrid-test.yaml name: test
Best Regards Pavan Kumar
On Thu, Feb 8, 2024 at 3:12 PM Milosz Wasilewski milosz.wasilewski@foundries.io wrote:
On Thu, Feb 8, 2024 at 3:15 AM pawanKumar pawan4749@gmail.com wrote:
Hi Milosz
Yes, I understand that.
But in my case I'm exporting results to bamboo using rest api. My issue is about failure retry case, if the action was failure for the first time it shows result as fail. Even though it is pass at second time, Rest API takes the first failure and displays as failure test result in bamboo.
In this case implementation of failure retry would be not useful. So, is there a way to override/ignore the first failure case.
Sounds like a bug, do you have an example job with this behaviour?
Best Regards, Milosz
Thanks.
On Wed, 7 Feb, 2024, 2:34 pm Milosz Wasilewski, milosz.wasilewski@foundries.io wrote:
Pawan,
I think you're confusing 2 different things. Job status only says if all actions in the job completed successfully. For example if your test script runs for too long, lava will call a timeout and end the job as 'incomplete'. Similarly if your infrastructure doesn't work properly (power_on command doesn't power the device on, etc). On the other hand test results can have 4 values: pass, fail, skip, unknown. The last one is only telling that something went wrong with data collection.
To sum up, failed test doesn't mean that the test job is incomplete. These are statuses of 2 different things.
Best Regards, Milosz
On Wed, Feb 7, 2024 at 4:08 AM pawanKumar pawan4749@gmail.com wrote:
Hi Team,
I was running the lava jobs and I found a difference in the status after job completion with failures. There were failures in i) The interactive test and lava overlay test [cases] but still the status of the job shows as "COMPLETE". ii) I tried an action failure retry, when action failure for the first time, it shows the result as failure but job status as complete. The retry- failure status should be ignored if it fails for the first time. I convert the test results into JUNIT using rest api. In the junit result it displays failure for the test case. My question is, How can we customize the status of the job or ignore the test result in case of failure-retry? Any Advice please.
Best Regards Pavan Kumar _______________________________________________ lava-devel mailing list -- lava-devel@lists.lavasoftware.org To unsubscribe send an email to lava-devel-leave@lists.lavasoftware.org
lava-devel@lists.lavasoftware.org
-
Milosz Wasilewski -
pawanKumar