I was too quick to hit reply. CCing lava-users for comments from LAVA team.

milosz

On 3 July 2017 at 21:50, Paul Sokolovsky paul.sokolovsky@linaro.org wrote:

Hello Milosz,

I appreciate getting at least some response ;-). Some questions however could use a reply from LAVA team, I guess.

On Mon, 3 Jul 2017 13:34:49 +0100 Milosz Wasilewski milosz.wasilewski@linaro.org wrote:

[]

...

...

jobs submit a number of tests to LAVA (via https://qa-reports.linaro.org/) for the following boards: arduino_101, frdm_k64f, frdm_kw41z, qemu_cortex_m3. Here's an example of cumulative test report for these platforms: https://qa-reports.linaro.org/lite/zephyr-upstream/tests/

That's really great! (Though the list of tests to run in LAVA seems to be hardcoded: https://git.linaro.org/ci/job/configs.git/tree/zephyr-upstream/submit_for_te...)

It is, as I wasn't really sure what to test. The build job needs to prepare the test templates to be submitted to LAVA. In case of zephyr each tests is a separate binary. So we end up with the number of file paths to substitute in the template. Hardcoding was the easiest thing to get things running. But I see no reason why it wouldn't be changed with some smarter code to discover the binaries. The problem with this approach is that some of these tests are just build time. They have no meaning when running on the board and need to be filter out somehow.

I see, that makes some sense. But thinking further, I'm not entirely sure about "build only" tests. Zephyr's sanitycheck test has such concept, but I'd imagine it comes from the following reasons: a) sanitycheck runs tests on QEMU, which has very bare hardware support, so many tests are not runnable; b) sanitycheck can operate on "samples", not just "tests", as sample can be interactive, etc. it makes sense to only build them, not run.

So, I'm not exactly sure about build-only tests on real HW boards. The "default" idea would be that they should run, but I imagine in reality, some may need to be filtered out. But then blacklisting would be better approach than whitelisting. And I'm not sure if Zephyr has concept of "skipped" tests which may be useful to handle hardware variations. (Well, I actually dunno if LAVA supports skipped tests!)

Anyway, these are rough ideas for the future. I've spent couple of weeks of munging with LITE CI setup, there're definitely some improvements, but also a Pandora box of other ideas and improvements to make. I'm wrapping up for now, but hope to look again in some time (definitely hope to look before the Connect, so we can discuss further steps there). In the meantime, I hope that more boards will be installed in the Lab and stability of them improves (so far they seem to be pretty flaky).

[]

...

...

• test: monitors:
  • name: foo start: "" end: Hello, ZJS world! pattern: (?P<result>(PASS|FAIL))\s-\s(?P<test_case_id>\w+).

So, the "start" substring is empty, and perhaps matches a line output by a USB multiplexer or board bootloader. "End" substring is actually the expected single-line output. And "pattern" is unused (dunno if it can be dropped without def file syntax error). Is there a better way to handle single-line test output?

You're making a silent assumption that if there is a matching line, the test is passed. In case of other tests (zephyr unit tests), it's not the case. The 'start' matches some line which is displayed when zephyr is booting. End matches the line which is displayed after all testing is done. The pattern follows the unit test pattern.

Thanks, but I'm not sure I understand this response. I don't challenge that Zephyr unittests need this support, or the way they're handled. LITE however needs to test more things than "batch" Zephyr unittests. I present another usercase which albeit simple, barely supported by LAVA. (That's a question to LAVA team definitely.)

...

...

Well, beyond a simple output matching, it would be nice even for the initial "smoke testing" to actually make some input into the application and check the expected output (e.g., input: "2+2", expected output: "4"). Is this already supported for LAVA "v2" pipeline tests? I may imagine that would be the same kind of functionality required to test bootloaders like U-boot for Linux boards.

I didn't use anything like this in v2 so far, but you're probably best off doing sth like

test 2+2=4 PASS.

than you can easily create pattern that will filter the output. In case of zephyr pattern is the only way to filter things out as there is no shell (?) on the board.

So, the problem, for starters, is how to make LAVA *feed* the input, as specified in the test definition (like "2+2") into a board.

As there were no reply from LAVA team (I may imagine they're busy with other things), I decided to create a user story in Jira for them, as I couldn't create a LAVA-* ticket, I created it as https://projects.linaro.org/browse/LITE-175 . Hopefully that won't go unnoticed and LAVA team would get to it eventually.

...

milosz

Thanks!

-- Best Regards, Paul

Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#%21/linaroorg - http://www.linaro.org/linaro-blog