Hello Milosz,

Thanks for routing this thread to lava-users - when I made initial post to linaro-validation, I check my archive and so that e.g. Neil posts there frequently, but I missed that it's not official LAVA list.

On Mon, 3 Jul 2017 22:25:31 +0100 Milosz Wasilewski milosz.wasilewski@linaro.org wrote:

[]

...

So, I'm not exactly sure about build-only tests on real HW boards. The "default" idea would be that they should run, but I imagine in reality, some may need to be filtered out. But then blacklisting would be better approach than whitelisting. And I'm not sure if Zephyr has concept of "skipped" tests which may be useful to handle hardware variations. (Well, I actually dunno if LAVA supports skipped tests!)

As far as I can tell they acutely run on the board, but usually output just 'Hello world!' or sth similar. As we discussed with Kumar, this is still OK. What Kumar requested (and I still didn't deliver) is that whenever the LAVA test job completes, the test should be considered 'passed'. So we wouldn't have to do any parsing of patterns. I'm not sure if that will work, but it's worth to try.

Hmm, I wonder what would be criteria for being "failed" for such tests... Anyway, thanks for sharing - I'm not familiar with all Zephyr tests/samples myself, will keep in mind such issues when looking into them.

[]

...

more boards will be installed in the Lab and stability of them improves (so far they seem to be pretty flaky).

You're absolutely right. This is a pretty big task to work on and IMHO requires someone to work full time at least for couple of weeks. The second part is also true, the boards don't behave as they should. I guess Dave can elaborate more on that. I can only see the result - boards (frdm-kw41z) don't run the tests they're requested.

Matt Hart actually showed me a ticket on that, so at least it's confirmed/known issue in works. But even with arduino_101 and frdm_k64f, I hit cases more than once when board(s) were stuck for extended time, but still were routed jobs to (which either failed or timed out). So, there may be problem with health checks, which either don't run frequently enough or aren't robust enough. arduino_101 is all the lone one, so if something happens to it, there's no backup. Etc, etc.

[]

...

So, the problem, for starters, is how to make LAVA *feed* the input, as specified in the test definition (like "2+2") into a board.

Right. What I proposed was coding all the inputs in the test itself.

Well, that would require bunch of legwork, but the biggest problem is that it wouldn't test what's actually required. E.g. both JerryScript and MicroPython Zephyr ports are actually interactive apps working over serial connection. And functional testing of them would be feeding something over this serial connection and checking that results are as expected. I'll keep in mind idea of "builtin" tests though.

Thanks!