Hello Neil,
On Tue, 4 Jul 2017 07:54:30 +0100 Neil Williams codehelp@debian.org wrote:
[]
Yes, LAVA has support for pass, fail, skip, unknown.
For POSIX shell tests, the test writer just calls lava-test-case name --result skip
For monitor tests, like Zephyr, it's down to the pattern but skip is as valid as pass and fail (as is unknown) for the result of the matches within the pattern.
Thanks, now that you said it, I remembered that some years ago I saw them all ;-). Thanks for confirming!
[]
There are known limitations with the USB subsystem and associated hardware across all architectures, affecting test devices and the workers which run the automation. LAVA has to drive that subsystem very hard for both fastboot devices and IoT devices. There are also problems due to the design of methods like fastboot and some of the IoT support which result from a single-developer model, leading to buggy performance when used at scale and added complexity in deploying workarounds to isolate such protocols in order to prevent interference between tests. The protocols themselves often lack robust error handling or retry support.
Yes, I understand all the complexity of it. But I'd hope that ability to power off both a device and USB hub connecting it would be the ultimate solution to such issues. Anyway, I guess this time, the question is more for the Lab team, not the LAVA team.
[]
LAVA result handling is ultimately a pattern matching system. Patterns must have a unique and reliable start string and a unique and reliable end string. An empty start string is just going to cause misleading results and bad pattern matches as the reality is that most boards emit some level of random junk immediately upon connection which needs to be ignored. So there needs to be a reliable, unique, start string emitted by the test device. It is not enough to *assume* a start at line zero, doing so increases the problems with reliability.
Good to know, and I generally agree. But with a real usecase of testing something which outputs just a single line, the only option is then to modify "device under test", which isn't always desirable and shows LAVA's inflexibility. Well, at least there's a workaround which works well enough, and hopefully will keep working ;-).
[]
That will need code changes, so please make a formal request for this support at CTT https://projects.linaro.org/servicedesk/customer/portal/1 so that we can track exactly what is required.
Thanks, now done as https://projects.linaro.org/servicedesk/customer/portal/1/CTT-413
[]