Hi All,
I am experimenting with the interactive test action, but cannot get it to work reliably.
I tried to replicate what is documented here: https://docs.lavasoftware.org/lava/interactive.html#writing-tests-interactiv... I also took as example a job mentioned previously on this mailing list: https://validation.linaro.org/scheduler/job/1925569/definition
But If I come down to the simplest uboot test job (yaml + log attached), I still see false verdicts. I first set 2 variables in uboot. Then I print them, and verify if the output contains the expected string. Only the first case behaves as expected: "fail-expected-1".
All following tests go wrong: Test case "fail-expected-2" fails, but for a wrong reason: "Matched a prompt (was expecting a success): '=> '" And remaining test cases also show a wrong verdict because of this. Including the last 2 TCs which are supposed to pass, but are marked as failed.
I also tested in Linux and on a different platform, and the behavior is the same.
Am I missing something with the syntax? Or with serial console settings?
Thanks a lot for your help,
Philippe Mazet
Hello,
I believe that you found a bug in the interactive test.
Looking at the logs we can see that:
2019-12-03T08:06:24 start: 3 lava-test-interactive-retry (timeout 00:01:00) [common] 2019-12-03T08:06:24 start: 3.1 lava-test-interactive (timeout 00:01:00) [common] 2019-12-03T08:06:24 Sending 'printenv testvariable1' 2019-12-03T08:06:24 printenv testvariable1 2019-12-03T08:06:24 u-boot=> printenv testvariable1 2019-12-03T08:06:24 Waiting for '=> ', '/ # ', 'testvalue1', 'inexistant_string' 2019-12-03T08:06:24 printenv testvariable1 2019-12-03T08:06:24 testvariable1=testvalue1 2019-12-03T08:06:24 Matched a failure: 'testvalue1'
The test is failing and lava is (because of a bug) not waiting for the prompt. So when sending the next command, lava will look at the DUT output and will see the previous prompt (the one that was not consumed in the previous step).
2019-12-03T08:06:24 {'case': 'fail-expected-1', 'definition': '0_interactive-test-verdict-check1', 'duration': '0.11', 'result': 'fail'} 2019-12-03T08:06:24 {'case': '0_interactive-test-verdict-check1', 'definition': 'lava', 'duration': '0.11', 'result': 'pass'} 2019-12-03T08:06:24 end: 3.1 lava-test-interactive (duration 00:00:00) [common] 2019-12-03T08:06:24 end: 3 lava-test-interactive-retry (duration 00:00:00) [common] 2019-12-03T08:06:24 start: 4 lava-test-interactive-retry (timeout 00:01:00) [common] 2019-12-03T08:06:24 start: 4.1 lava-test-interactive (timeout 00:01:00) [common] 2019-12-03T08:06:24 Sending 'printenv testvariable2' 2019-12-03T08:06:24 printenv testvariable2 2019-12-03T08:06:24 u-boot=> printenv testvariable2 2019-12-03T08:06:24 Waiting for '=> ', '/ # ', 'testvalue2', 'testvalue2' 2019-12-03T08:06:24 Matched a prompt (was expecting a success): '=> '
Here lava is (wrongly) matching the previous prompt and hence marking the test as failed.
I will submit an issue in the bug tracker and send a merge request to fix this. Are you able to patch your lava instance to test merge requests?
Thanks
Le mar. 3 déc. 2019 à 09:18, Philippe Mazet philippe.mazet@nxp.com a écrit :
Hi All,
I am experimenting with the interactive test action, but cannot get it to work reliably.
I tried to replicate what is documented here: https://docs.lavasoftware.org/lava/interactive.html#writing-tests-interactiv...
I also took as example a job mentioned previously on this mailing list: https://validation.linaro.org/scheduler/job/1925569/definition
But If I come down to the simplest uboot test job (yaml + log attached), I still see false verdicts.
I first set 2 variables in uboot. Then I print them, and verify if the output contains the expected string.
Only the first case behaves as expected: “fail-expected-1”.
All following tests go wrong:
Test case “fail-expected-2” fails, but for a wrong reason: “Matched a prompt (was expecting a success): '=> '”
And remaining test cases also show a wrong verdict because of this. Including the last 2 TCs which are supposed to pass, but are marked as failed.
I also tested in Linux and on a different platform, and the behavior is the same.
Am I missing something with the syntax? Or with serial console settings?
Thanks a lot for your help,
Philippe Mazet
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
I created issue https://git.lavasoftware.org/lava/lava/issues/360
Thanks
Le mar. 3 déc. 2019 à 17:34, Remi Duraffort remi.duraffort@linaro.org a écrit :
Hello,
I believe that you found a bug in the interactive test.
Looking at the logs we can see that:
2019-12-03T08:06:24 start: 3 lava-test-interactive-retry (timeout 00:01:00) [common] 2019-12-03T08:06:24 start: 3.1 lava-test-interactive (timeout 00:01:00) [common] 2019-12-03T08:06:24 Sending 'printenv testvariable1' 2019-12-03T08:06:24 printenv testvariable1 2019-12-03T08:06:24 u-boot=> printenv testvariable1 2019-12-03T08:06:24 Waiting for '=> ', '/ # ', 'testvalue1', 'inexistant_string' 2019-12-03T08:06:24 printenv testvariable1 2019-12-03T08:06:24 testvariable1=testvalue1 2019-12-03T08:06:24 Matched a failure: 'testvalue1'
The test is failing and lava is (because of a bug) not waiting for the prompt. So when sending the next command, lava will look at the DUT output and will see the previous prompt (the one that was not consumed in the previous step).
2019-12-03T08:06:24 {'case': 'fail-expected-1', 'definition': '0_interactive-test-verdict-check1', 'duration': '0.11', 'result': 'fail'} 2019-12-03T08:06:24 {'case': '0_interactive-test-verdict-check1', 'definition': 'lava', 'duration': '0.11', 'result': 'pass'} 2019-12-03T08:06:24 end: 3.1 lava-test-interactive (duration 00:00:00) [common] 2019-12-03T08:06:24 end: 3 lava-test-interactive-retry (duration 00:00:00) [common] 2019-12-03T08:06:24 start: 4 lava-test-interactive-retry (timeout 00:01:00) [common] 2019-12-03T08:06:24 start: 4.1 lava-test-interactive (timeout 00:01:00) [common] 2019-12-03T08:06:24 Sending 'printenv testvariable2' 2019-12-03T08:06:24 printenv testvariable2 2019-12-03T08:06:24 u-boot=> printenv testvariable2 2019-12-03T08:06:24 Waiting for '=> ', '/ # ', 'testvalue2', 'testvalue2' 2019-12-03T08:06:24 Matched a prompt (was expecting a success): '=> '
Here lava is (wrongly) matching the previous prompt and hence marking the test as failed.
I will submit an issue in the bug tracker and send a merge request to fix this. Are you able to patch your lava instance to test merge requests?
Thanks
Le mar. 3 déc. 2019 à 09:18, Philippe Mazet philippe.mazet@nxp.com a écrit :
Hi All,
I am experimenting with the interactive test action, but cannot get it to work reliably.
I tried to replicate what is documented here: https://docs.lavasoftware.org/lava/interactive.html#writing-tests-interactiv...
I also took as example a job mentioned previously on this mailing list: https://validation.linaro.org/scheduler/job/1925569/definition
But If I come down to the simplest uboot test job (yaml + log attached), I still see false verdicts.
I first set 2 variables in uboot. Then I print them, and verify if the output contains the expected string.
Only the first case behaves as expected: “fail-expected-1”.
All following tests go wrong:
Test case “fail-expected-2” fails, but for a wrong reason: “Matched a prompt (was expecting a success): '=> '”
And remaining test cases also show a wrong verdict because of this. Including the last 2 TCs which are supposed to pass, but are marked as failed.
I also tested in Linux and on a different platform, and the behavior is the same.
Am I missing something with the syntax? Or with serial console settings?
Thanks a lot for your help,
Philippe Mazet
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
-- Rémi Duraffort LAVA Architect Linaro
Great, thanks a lot!
And yes, I can patch my staging instance and test your fix when available.
Philippe
From: Remi Duraffort remi.duraffort@linaro.org Sent: Tuesday, December 3, 2019 5:39 PM To: Philippe Mazet philippe.mazet@nxp.com Cc: lava-users@lists.lavasoftware.org lava-users@lavasoftware.org Subject: [EXT] Re: [Lava-users] Interactive test action unreliable behavior
Caution: EXT Email I created issue https://git.lavasoftware.org/lava/lava/issues/360https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.lavasoftware.org%2Flava%2Flava%2Fissues%2F360&data=02%7C01%7Cphilippe.mazet%40nxp.com%7Ccf90992998984887692c08d7780f585f%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637109879612531140&sdata=ChuWa2GWjLRvdOEXK5I4PNZb1YMAXmj%2FDt4BwZmgoKc%3D&reserved=0
Thanks
Le mar. 3 déc. 2019 à 17:34, Remi Duraffort <remi.duraffort@linaro.orgmailto:remi.duraffort@linaro.org> a écrit : Hello,
I believe that you found a bug in the interactive test.
Looking at the logs we can see that:
2019-12-03T08:06:24 start: 3 lava-test-interactive-retry (timeout 00:01:00) [common] 2019-12-03T08:06:24 start: 3.1 lava-test-interactive (timeout 00:01:00) [common] 2019-12-03T08:06:24 Sending 'printenv testvariable1' 2019-12-03T08:06:24 printenv testvariable1 2019-12-03T08:06:24 u-boot=> printenv testvariable1 2019-12-03T08:06:24 Waiting for '=> ', '/ # ', 'testvalue1', 'inexistant_string' 2019-12-03T08:06:24 printenv testvariable1 2019-12-03T08:06:24 testvariable1=testvalue1 2019-12-03T08:06:24 Matched a failure: 'testvalue1'
The test is failing and lava is (because of a bug) not waiting for the prompt. So when sending the next command, lava will look at the DUT output and will see the previous prompt (the one that was not consumed in the previous step).
2019-12-03T08:06:24 {'case': 'fail-expected-1', 'definition': '0_interactive-test-verdict-check1', 'duration': '0.11', 'result': 'fail'} 2019-12-03T08:06:24 {'case': '0_interactive-test-verdict-check1', 'definition': 'lava', 'duration': '0.11', 'result': 'pass'} 2019-12-03T08:06:24 end: 3.1 lava-test-interactive (duration 00:00:00) [common] 2019-12-03T08:06:24 end: 3 lava-test-interactive-retry (duration 00:00:00) [common] 2019-12-03T08:06:24 start: 4 lava-test-interactive-retry (timeout 00:01:00) [common] 2019-12-03T08:06:24 start: 4.1 lava-test-interactive (timeout 00:01:00) [common] 2019-12-03T08:06:24 Sending 'printenv testvariable2' 2019-12-03T08:06:24 printenv testvariable2 2019-12-03T08:06:24 u-boot=> printenv testvariable2 2019-12-03T08:06:24 Waiting for '=> ', '/ # ', 'testvalue2', 'testvalue2' 2019-12-03T08:06:24 Matched a prompt (was expecting a success): '=> '
Here lava is (wrongly) matching the previous prompt and hence marking the test as failed.
I will submit an issue in the bug tracker and send a merge request to fix this. Are you able to patch your lava instance to test merge requests?
Thanks
Le mar. 3 déc. 2019 à 09:18, Philippe Mazet <philippe.mazet@nxp.commailto:philippe.mazet@nxp.com> a écrit : Hi All,
I am experimenting with the interactive test action, but cannot get it to work reliably.
I tried to replicate what is documented here: https://docs.lavasoftware.org/lava/interactive.html#writing-tests-interactiv...https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.lavasoftware.org%2Flava%2Finteractive.html%23writing-tests-interactive&data=02%7C01%7Cphilippe.mazet%40nxp.com%7Ccf90992998984887692c08d7780f585f%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637109879612531140&sdata=a7Cm9YXcjPSpgx5IrK0BR8k1VQBhZip9xJHKTc4wAFQ%3D&reserved=0 I also took as example a job mentioned previously on this mailing list: https://validation.linaro.org/scheduler/job/1925569/definitionhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fvalidation.linaro.org%2Fscheduler%2Fjob%2F1925569%2Fdefinition&data=02%7C01%7Cphilippe.mazet%40nxp.com%7Ccf90992998984887692c08d7780f585f%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637109879612541132&sdata=6VjVz%2BpSUUVFe2xKKy1K7y%2BVd3DRwz8YOrOo1syW8sg%3D&reserved=0
But If I come down to the simplest uboot test job (yaml + log attached), I still see false verdicts. I first set 2 variables in uboot. Then I print them, and verify if the output contains the expected string. Only the first case behaves as expected: “fail-expected-1”.
All following tests go wrong: Test case “fail-expected-2” fails, but for a wrong reason: “Matched a prompt (was expecting a success): '=> '” And remaining test cases also show a wrong verdict because of this. Including the last 2 TCs which are supposed to pass, but are marked as failed.
I also tested in Linux and on a different platform, and the behavior is the same.
Am I missing something with the syntax? Or with serial console settings?
Thanks a lot for your help,
Philippe Mazet
_______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.orgmailto:Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-usershttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.lavasoftware.org%2Fmailman%2Flistinfo%2Flava-users&data=02%7C01%7Cphilippe.mazet%40nxp.com%7Ccf90992998984887692c08d7780f585f%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637109879612541132&sdata=kCvEFMQPX1TvDr6LQssognA%2Fjka1U2anDh1i%2BZ7f9IQ%3D&reserved=0
-- Rémi Duraffort LAVA Architect Linaro
-- Rémi Duraffort LAVA Architect Linaro
lava-users@lists.lavasoftware.org
-
Philippe Mazet -
Remi Duraffort