Hello,

I believe that you found a bug in the interactive test.

Looking at the logs we can see that:

2019-12-03T08:06:24 start: 3 lava-test-interactive-retry (timeout 00:01:00) [common]
2019-12-03T08:06:24 start: 3.1 lava-test-interactive (timeout 00:01:00) [common]
2019-12-03T08:06:24 Sending 'printenv testvariable1'
2019-12-03T08:06:24 printenv testvariable1
2019-12-03T08:06:24 u-boot=> printenv testvariable1
2019-12-03T08:06:24 Waiting for '=> ', '/ # ', 'testvalue1', 'inexistant_string'
2019-12-03T08:06:24 printenv testvariable1
2019-12-03T08:06:24 testvariable1=testvalue1
2019-12-03T08:06:24 Matched a failure: 'testvalue1'

The test is failing and lava is (because of a bug) not waiting for the prompt. So when sending the next command, lava will look at the DUT output and will see the previous prompt (the one that was not consumed in the previous step).

2019-12-03T08:06:24 {'case': 'fail-expected-1', 'definition': '0_interactive-test-verdict-check1', 'duration': '0.11', 'result': 'fail'}
2019-12-03T08:06:24 {'case': '0_interactive-test-verdict-check1', 'definition': 'lava', 'duration': '0.11', 'result': 'pass'}
2019-12-03T08:06:24 end: 3.1 lava-test-interactive (duration 00:00:00) [common]
2019-12-03T08:06:24 end: 3 lava-test-interactive-retry (duration 00:00:00) [common]
2019-12-03T08:06:24 start: 4 lava-test-interactive-retry (timeout 00:01:00) [common]
2019-12-03T08:06:24 start: 4.1 lava-test-interactive (timeout 00:01:00) [common]
2019-12-03T08:06:24 Sending 'printenv testvariable2'
2019-12-03T08:06:24 printenv testvariable2
2019-12-03T08:06:24 u-boot=> printenv testvariable2
2019-12-03T08:06:24 Waiting for '=> ', '/ # ', 'testvalue2', 'testvalue2'
2019-12-03T08:06:24 Matched a prompt (was expecting a success): '=> '

Here lava is (wrongly) matching the previous prompt and hence marking the test as failed.

I will submit an issue in the bug tracker and send a merge request to fix this. Are you able to patch your lava instance to test merge requests?

Thanks

Le mar. 3 déc. 2019 à 09:18, Philippe Mazet <philippe.mazet@nxp.com> a écrit :

Hi All,

I am experimenting with the interactive test action, but cannot get it to work reliably.

I tried to replicate what is documented here: https://docs.lavasoftware.org/lava/interactive.html#writing-tests-interactive

I also took as example a job mentioned previously on this mailing list: https://validation.linaro.org/scheduler/job/1925569/definition

But If I come down to the simplest uboot test job (yaml + log attached), I still see false verdicts.

I first set 2 variables in uboot. Then I print them, and verify if the output contains the expected string.

Only the first case behaves as expected: “fail-expected-1”.

All following tests go wrong:

Test case “fail-expected-2” fails, but for a wrong reason: “Matched a prompt (was expecting a success): '=> '”

And remaining test cases also show a wrong verdict because of this. Including the last 2 TCs which are supposed to pass, but are marked as failed.

I also tested in Linux and on a different platform, and the behavior is the same.

Am I missing something with the syntax? Or with serial console settings?

Thanks a lot for your help,

Philippe Mazet

_______________________________________________
Lava-users mailing list
Lava-users@lists.lavasoftware.org
https://lists.lavasoftware.org/mailman/listinfo/lava-users